Nov 03 07:52:40.019022 ip-10-0-140-123 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 07:52:40.427595 ip-10-0-140-123 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 07:52:40.427595 ip-10-0-140-123 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 07:52:40.427595 ip-10-0-140-123 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 07:52:40.427595 ip-10-0-140-123 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 07:52:40.431809 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431698    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:52:40.431809 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431808    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:52:40.431809 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431814    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:52:40.432013 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431819    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:52:40.432013 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431823    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:52:40.432013 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431827    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:52:40.432013 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431831    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:52:40.432013 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431837    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:52:40.432013 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431840    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:52:40.432013 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431844    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:52:40.432013 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431848    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:52:40.432013 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431852    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:52:40.432013 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431856    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:52:40.432013 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431859    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:52:40.432013 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431863    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:52:40.432013 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431867    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:52:40.432013 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431871    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:52:40.432013 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431876    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:52:40.432013 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431880    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:52:40.432013 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431884    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:52:40.432013 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431888    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:52:40.432013 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431891    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:52:40.432793 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431895    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:52:40.432793 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431899    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:52:40.432793 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431903    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:52:40.432793 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431906    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:52:40.432793 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431910    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:52:40.432793 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431914    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:52:40.432793 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431918    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:52:40.432793 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431922    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:52:40.432793 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431926    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:52:40.432793 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431930    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:52:40.432793 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431948    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:52:40.432793 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431953    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:52:40.432793 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431957    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:52:40.432793 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431961    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:52:40.432793 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431966    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:52:40.432793 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431970    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:52:40.432793 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431975    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:52:40.432793 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431979    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:52:40.432793 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431983    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:52:40.433579 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431988    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:52:40.433579 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431992    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:52:40.433579 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.431996    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:52:40.433579 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432000    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:52:40.433579 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432005    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:52:40.433579 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432009    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:52:40.433579 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432013    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:52:40.433579 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432017    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:52:40.433579 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432021    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:52:40.433579 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432026    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:52:40.433579 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432030    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:52:40.433579 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432034    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:52:40.433579 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432039    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:52:40.433579 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432043    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:52:40.433579 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432047    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:52:40.433579 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432052    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:52:40.433579 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432056    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:52:40.433579 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432061    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:52:40.433579 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432065    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:52:40.433579 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432070    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:52:40.434449 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432074    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:52:40.434449 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432078    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:52:40.434449 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432083    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:52:40.434449 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432087    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:52:40.434449 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432091    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:52:40.434449 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432097    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:52:40.434449 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432102    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:52:40.434449 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432107    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:52:40.434449 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432112    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:52:40.434449 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432116    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:52:40.434449 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432120    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:52:40.434449 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432127    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:52:40.434449 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432134    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:52:40.434449 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432140    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:52:40.434449 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432144    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:52:40.434449 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432149    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:52:40.434449 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432153    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:52:40.434449 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432157    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:52:40.434449 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432161    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:52:40.434449 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432165    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:52:40.435362 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432170    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:52:40.435362 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432174    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:52:40.435362 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432181    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:52:40.435362 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432185    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:52:40.435362 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432189    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:52:40.435362 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432193    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:52:40.435362 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432197    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:52:40.435362 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432202    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:52:40.435362 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432206    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:52:40.435362 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432211    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:52:40.435362 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432215    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:52:40.435362 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432219    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:52:40.435362 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432223    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:52:40.435362 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432228    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:52:40.435362 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432234    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:52:40.435362 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432238    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:52:40.435362 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432242    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:52:40.435362 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432246    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:52:40.435362 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432251    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:52:40.435362 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432256    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:52:40.436225 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432260    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:52:40.436225 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432265    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:52:40.436225 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432269    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:52:40.436225 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432273    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:52:40.436225 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.432277    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:52:40.436225 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433538    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:52:40.436225 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433548    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:52:40.436225 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433552    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:52:40.436225 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433557    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:52:40.436225 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433561    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:52:40.436225 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433566    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:52:40.436225 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433570    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:52:40.436225 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433574    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:52:40.436225 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433578    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:52:40.436225 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433582    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:52:40.436225 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433588    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:52:40.436225 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433592    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:52:40.436225 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433596    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:52:40.436225 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433600    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:52:40.436225 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433604    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:52:40.437161 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433611    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:52:40.437161 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433619    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:52:40.437161 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433624    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:52:40.437161 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433628    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:52:40.437161 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433634    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:52:40.437161 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433638    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:52:40.437161 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433642    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:52:40.437161 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433646    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:52:40.437161 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433650    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:52:40.437161 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433654    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:52:40.437161 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433659    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:52:40.437161 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433663    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:52:40.437161 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433668    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:52:40.437161 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433672    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:52:40.437161 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433676    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:52:40.437161 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433680    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:52:40.437161 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433685    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:52:40.437161 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433689    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:52:40.437161 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433692    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:52:40.437910 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433696    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:52:40.437910 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433701    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:52:40.437910 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433705    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:52:40.437910 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433709    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:52:40.437910 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433714    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:52:40.437910 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433718    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:52:40.437910 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433722    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:52:40.437910 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433726    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:52:40.437910 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433730    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:52:40.437910 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433735    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:52:40.437910 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433740    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:52:40.437910 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433744    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:52:40.437910 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433748    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:52:40.437910 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433752    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:52:40.437910 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433774    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:52:40.437910 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433779    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:52:40.437910 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433783    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:52:40.437910 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433787    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:52:40.437910 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433792    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:52:40.438599 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433796    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:52:40.438599 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433806    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:52:40.438599 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433810    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:52:40.438599 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433814    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:52:40.438599 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433818    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:52:40.438599 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433823    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:52:40.438599 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433827    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:52:40.438599 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433832    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:52:40.438599 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433836    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:52:40.438599 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433840    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:52:40.438599 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433844    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:52:40.438599 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433848    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:52:40.438599 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433853    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:52:40.438599 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433857    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:52:40.438599 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433861    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:52:40.438599 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433865    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:52:40.438599 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433870    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:52:40.438599 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433874    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:52:40.438599 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433878    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:52:40.439089 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433882    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:52:40.439089 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433886    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:52:40.439089 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433891    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:52:40.439089 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433895    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:52:40.439089 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433910    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:52:40.439089 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433915    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:52:40.439089 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433919    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:52:40.439089 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433923    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:52:40.439089 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433930    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:52:40.439089 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433935    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:52:40.439089 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433941    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:52:40.439089 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433947    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:52:40.439089 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433952    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:52:40.439089 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433956    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:52:40.439089 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433960    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:52:40.439089 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433965    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:52:40.439089 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433970    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:52:40.439089 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433973    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:52:40.439089 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433979    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:52:40.439539 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433983    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:52:40.439539 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433988    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:52:40.439539 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433992    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:52:40.439539 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.433996    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:52:40.439539 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434000    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:52:40.439539 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434004    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:52:40.439539 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434010    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:52:40.439539 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434015    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:52:40.439539 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434019    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:52:40.439539 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434023    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:52:40.439539 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434027    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:52:40.439539 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434031    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:52:40.439539 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434036    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:52:40.439539 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434040    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:52:40.439539 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434044    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:52:40.439539 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434855    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:52:40.439539 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434864    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:52:40.439539 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434870    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:52:40.439539 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434875    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:52:40.439539 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434879    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:52:40.440116 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434884    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:52:40.440116 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434888    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:52:40.440116 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434892    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:52:40.440116 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434897    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:52:40.440116 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434902    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:52:40.440116 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434907    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:52:40.440116 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434911    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:52:40.440116 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434916    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:52:40.440116 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434920    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:52:40.440116 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434924    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:52:40.440116 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434928    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:52:40.440116 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434933    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:52:40.440116 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434937    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:52:40.440116 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434944    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:52:40.440116 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434951    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:52:40.440116 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434957    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:52:40.440116 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434962    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:52:40.440116 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434967    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:52:40.440116 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434972    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:52:40.440566 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434976    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:52:40.440566 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434981    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:52:40.440566 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434986    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:52:40.440566 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434990    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:52:40.440566 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434995    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:52:40.440566 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.434999    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:52:40.440566 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435003    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:52:40.440566 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435007    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:52:40.440566 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435011    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:52:40.440566 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435016    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:52:40.440566 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435020    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:52:40.440566 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435025    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:52:40.440566 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435033    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:52:40.440566 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435038    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:52:40.440566 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435042    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:52:40.440566 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435046    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:52:40.440566 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435050    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:52:40.440566 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435054    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:52:40.440566 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435058    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:52:40.440566 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435062    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:52:40.441072 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435066    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:52:40.441072 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435070    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:52:40.441072 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435075    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:52:40.441072 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435079    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:52:40.441072 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435083    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:52:40.441072 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435087    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:52:40.441072 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435091    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:52:40.441072 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435095    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:52:40.441072 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435099    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:52:40.441072 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435103    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:52:40.441072 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435107    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:52:40.441072 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435113    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:52:40.441072 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435118    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:52:40.441072 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435122    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:52:40.441072 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435126    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:52:40.441072 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435130    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:52:40.441072 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435135    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:52:40.441072 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435140    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:52:40.441072 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435144    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:52:40.441522 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435148    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:52:40.441522 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435151    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:52:40.441522 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435155    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:52:40.441522 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435159    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:52:40.441522 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435163    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:52:40.441522 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435171    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:52:40.441522 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435175    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:52:40.441522 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435180    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:52:40.441522 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435184    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:52:40.441522 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435188    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:52:40.441522 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435192    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:52:40.441522 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435196    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:52:40.441522 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435200    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:52:40.441522 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435205    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:52:40.441522 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435209    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:52:40.441522 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435213    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:52:40.441522 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435217    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:52:40.441522 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435221    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:52:40.441522 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435225    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:52:40.442063 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435230    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:52:40.442063 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435234    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:52:40.442063 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435238    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:52:40.442063 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435242    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:52:40.442063 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435246    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:52:40.442063 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435252    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:52:40.442063 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435256    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:52:40.442063 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435260    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:52:40.442063 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435264    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:52:40.442063 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435268    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:52:40.442063 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435274    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:52:40.442063 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435279    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:52:40.442063 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435284    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:52:40.442063 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435288    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:52:40.442063 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435293    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:52:40.442063 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435297    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:52:40.442063 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435301    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:52:40.442063 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435305    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:52:40.442063 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435312    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435317    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435321    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435325    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435330    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.435335    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435483    2579 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435494    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435508    2579 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435515    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435521    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435526    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435534    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435540    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435545    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435550    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435556    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435561    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435566    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435572    2579 flags.go:64] FLAG: --cgroup-root=""
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435576    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435581    2579 flags.go:64] FLAG: --client-ca-file=""
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435586    2579 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 07:52:40.442556 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435590    2579 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435596    2579 flags.go:64] FLAG: --cluster-domain=""
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435601    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435606    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435611    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435616    2579 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435622    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435627    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435632    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435640    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435645    2579 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435650    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435655    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435660    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435665    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435672    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435677    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435682    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435687    2579 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435691    2579 flags.go:64] FLAG: --enable-server="true"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435696    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435703    2579 flags.go:64] FLAG: --event-burst="100"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435708    2579 flags.go:64] FLAG: --event-qps="50"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435713    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 07:52:40.443129 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435718    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435723    2579 flags.go:64] FLAG: --eviction-hard=""
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435729    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435734    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435740    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435744    2579 flags.go:64] FLAG: --eviction-soft=""
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435749    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435772    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435777    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435782    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435787    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435791    2579 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435796    2579 flags.go:64] FLAG: --feature-gates=""
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435802    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435807    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435812    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435817    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435824    2579 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435829    2579 flags.go:64] FLAG: --help="false"
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435834    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435840    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435845    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435850    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 07:52:40.443703 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435855    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435861    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435866    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435871    2579 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435876    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435880    2579 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435885    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435890    2579 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435895    2579 flags.go:64] FLAG: --kube-reserved=""
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435900    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435905    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435910    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435915    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435920    2579 flags.go:64] FLAG: --lock-file=""
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435924    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435929    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435934    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435943    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435948    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435953    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435958    2579 flags.go:64] FLAG: --logging-format="text"
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435962    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435968    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435972    2579 flags.go:64] FLAG: --manifest-url=""
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435976    2579 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 07:52:40.444286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435984    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435990    2579 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.435996    2579 flags.go:64] FLAG: --max-pods="110"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436001    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436006    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436011    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436016    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436021    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436026    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436031    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436042    2579 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436047    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436051    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436056    2579 flags.go:64] FLAG: --pod-cidr=""
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436061    2579 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436065    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436070    2579 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436075    2579 flags.go:64] FLAG: --port="10250"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436080    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436084    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0c05fecb68435fa38"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436090    2579 flags.go:64] FLAG: --qos-reserved=""
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436095    2579 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436100    2579 flags.go:64] FLAG: --register-node="true"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436104    2579 flags.go:64] FLAG: --register-with-taints=""
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436110    2579 flags.go:64] FLAG: --registry-burst="10"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436115    2579 flags.go:64] FLAG: --registry-qps="5"
Nov 03 07:52:40.444913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436120    2579 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436124    2579 flags.go:64] FLAG: --reserved-memory=""
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436130    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436135    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436140    2579 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436144    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436149    2579 flags.go:64] FLAG: --runonce="false"
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436156    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436161    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436166    2579 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436171    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436175    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436181    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436186    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436191    2579 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436196    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436200    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436205    2579 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436212    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436217    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436221    2579 flags.go:64] FLAG: --system-cgroups=""
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436226    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436234    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436238    2579 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436242    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 07:52:40.445527 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436248    2579 flags.go:64] FLAG: --tls-min-version=""
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436253    2579 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436258    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436262    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436268    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436272    2579 flags.go:64] FLAG: --v="2"
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436279    2579 flags.go:64] FLAG: --version="false"
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436285    2579 flags.go:64] FLAG: --vmodule=""
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436292    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.436298    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436510    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436517    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436521    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436526    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436532    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436537    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436541    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436545    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436550    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436554    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436560    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436564    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:52:40.446148 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436569    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:52:40.446674 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436573    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:52:40.446674 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436578    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:52:40.446674 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436584    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:52:40.446674 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436589    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:52:40.446674 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436595    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:52:40.446674 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436601    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:52:40.446674 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436605    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:52:40.446674 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436609    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:52:40.446674 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436614    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:52:40.446674 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436618    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:52:40.446674 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436622    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:52:40.446674 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436626    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:52:40.446674 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436631    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:52:40.446674 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436635    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:52:40.446674 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436640    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:52:40.446674 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436644    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:52:40.446674 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436648    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:52:40.446674 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436652    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:52:40.446674 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436657    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:52:40.447142 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436661    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:52:40.447142 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436665    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:52:40.447142 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436670    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:52:40.447142 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436674    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:52:40.447142 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436680    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:52:40.447142 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436684    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:52:40.447142 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436689    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:52:40.447142 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436693    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:52:40.447142 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436698    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:52:40.447142 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436703    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:52:40.447142 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436707    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:52:40.447142 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436712    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:52:40.447142 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436716    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:52:40.447142 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436721    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:52:40.447142 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436725    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:52:40.447142 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436731    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:52:40.447142 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436736    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:52:40.447142 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436740    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:52:40.447142 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436744    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:52:40.447615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436748    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:52:40.447615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436753    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:52:40.447615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436773    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:52:40.447615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436778    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:52:40.447615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436782    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:52:40.447615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436786    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:52:40.447615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436789    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:52:40.447615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436794    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:52:40.447615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436798    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:52:40.447615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436802    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:52:40.447615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436806    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:52:40.447615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436810    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:52:40.447615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436814    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:52:40.447615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436818    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:52:40.447615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436822    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:52:40.447615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436827    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:52:40.447615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436834    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:52:40.447615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436843    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:52:40.447615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436848    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:52:40.447615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436852    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:52:40.448104 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436857    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:52:40.448104 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436861    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:52:40.448104 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436865    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:52:40.448104 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436869    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:52:40.448104 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436874    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:52:40.448104 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436879    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:52:40.448104 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436884    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:52:40.448104 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436888    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:52:40.448104 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436894    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:52:40.448104 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436899    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:52:40.448104 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436903    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:52:40.448104 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436907    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:52:40.448104 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436911    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:52:40.448104 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436915    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:52:40.448104 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436919    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:52:40.448104 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436923    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:52:40.448104 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436928    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:52:40.448104 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436932    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:52:40.448104 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436936    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:52:40.448537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436940    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:52:40.448537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436945    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:52:40.448537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436949    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:52:40.448537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436953    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:52:40.448537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436957    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:52:40.448537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436961    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:52:40.448537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436966    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:52:40.448537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436970    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:52:40.448537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436975    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:52:40.448537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436979    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:52:40.448537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436984    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:52:40.448537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436989    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:52:40.448537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436993    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:52:40.448537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.436997    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:52:40.448537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437001    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:52:40.448537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437007    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:52:40.448963 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.437050    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 07:52:40.448963 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437113    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:52:40.448963 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437121    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:52:40.448963 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437126    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:52:40.448963 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437132    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:52:40.448963 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437137    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:52:40.448963 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437145    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:52:40.448963 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437149    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:52:40.448963 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437153    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:52:40.448963 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437157    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:52:40.448963 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437162    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:52:40.448963 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437166    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:52:40.448963 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437171    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:52:40.448963 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437177    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:52:40.448963 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437182    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:52:40.448963 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437187    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:52:40.449341 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437191    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:52:40.449341 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437195    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:52:40.449341 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437199    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:52:40.449341 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437204    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:52:40.449341 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437208    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:52:40.449341 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437213    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:52:40.449341 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437217    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:52:40.449341 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437221    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:52:40.449341 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437226    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:52:40.449341 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437230    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:52:40.449341 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437234    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:52:40.449341 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437239    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:52:40.449341 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437243    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:52:40.449341 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437247    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:52:40.449341 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437253    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:52:40.449341 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437260    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:52:40.449341 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437265    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:52:40.449341 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437269    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:52:40.449341 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437275    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:52:40.449341 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437279    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:52:40.449837 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437284    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:52:40.449837 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437289    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:52:40.449837 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437295    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:52:40.449837 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437299    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:52:40.449837 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437304    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:52:40.449837 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437308    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:52:40.449837 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437312    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:52:40.449837 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437316    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:52:40.449837 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437320    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:52:40.449837 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437325    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:52:40.449837 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437329    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:52:40.449837 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437333    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:52:40.449837 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437337    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:52:40.449837 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437342    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:52:40.449837 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437346    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:52:40.449837 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437350    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:52:40.449837 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437354    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:52:40.449837 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437358    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:52:40.449837 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437364    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:52:40.449837 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437369    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:52:40.450301 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437373    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:52:40.450301 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437377    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:52:40.450301 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437382    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:52:40.450301 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437386    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:52:40.450301 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437390    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:52:40.450301 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437394    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:52:40.450301 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437398    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:52:40.450301 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437403    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:52:40.450301 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437407    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:52:40.450301 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437412    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:52:40.450301 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437416    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:52:40.450301 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437420    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:52:40.450301 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437425    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:52:40.450301 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437430    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:52:40.450301 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437436    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:52:40.450301 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437440    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:52:40.450301 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437444    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:52:40.450301 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437448    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:52:40.450301 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437453    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:52:40.450301 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437457    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:52:40.450803 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437461    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:52:40.450803 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437465    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:52:40.450803 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437469    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:52:40.450803 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437474    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:52:40.450803 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437478    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:52:40.450803 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437482    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:52:40.450803 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437486    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:52:40.450803 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437490    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:52:40.450803 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437494    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:52:40.450803 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437499    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:52:40.450803 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437503    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:52:40.450803 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437507    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:52:40.450803 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437514    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:52:40.450803 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437518    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:52:40.450803 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437522    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:52:40.450803 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437527    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:52:40.450803 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437531    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:52:40.450803 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437535    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:52:40.450803 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437539    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:52:40.451247 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437544    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:52:40.451247 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437548    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:52:40.451247 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437552    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:52:40.451247 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437557    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:52:40.451247 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437561    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:52:40.451247 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437565    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:52:40.451247 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437569    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:52:40.451247 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437576    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:52:40.451247 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437580    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:52:40.451247 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437585    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:52:40.451247 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437589    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:52:40.451247 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.437593    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:52:40.451247 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.443975    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 07:52:40.451247 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.443990    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 07:52:40.451247 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.444005    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 07:52:40.451247 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.444009    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 07:52:40.451247 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444061    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:52:40.451247 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444066    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:52:40.451247 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444069    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:52:40.451247 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444072    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:52:40.451723 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444076    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:52:40.451723 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444078    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:52:40.451723 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444081    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:52:40.451723 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444084    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:52:40.451723 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444086    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:52:40.451723 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444089    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:52:40.451723 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444092    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:52:40.451723 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444095    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:52:40.451723 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444098    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:52:40.451723 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444101    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:52:40.451723 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444103    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:52:40.451723 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444106    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:52:40.451723 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444108    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:52:40.451723 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444111    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:52:40.451723 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444114    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:52:40.451723 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444117    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:52:40.451723 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444119    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:52:40.451723 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444122    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:52:40.451723 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444125    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:52:40.451723 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444128    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:52:40.452208 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444131    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:52:40.452208 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444133    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:52:40.452208 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444137    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:52:40.452208 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444139    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:52:40.452208 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444142    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:52:40.452208 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444145    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:52:40.452208 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444148    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:52:40.452208 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444150    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:52:40.452208 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444153    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:52:40.452208 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444156    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:52:40.452208 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444158    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:52:40.452208 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444161    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:52:40.452208 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444163    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:52:40.452208 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444166    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:52:40.452208 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444168    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:52:40.452208 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444171    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:52:40.452208 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444174    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:52:40.452208 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444176    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:52:40.452208 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444179    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:52:40.452208 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444181    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:52:40.452751 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444184    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:52:40.452751 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444186    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:52:40.452751 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444189    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:52:40.452751 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444191    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:52:40.452751 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444194    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:52:40.452751 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444197    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:52:40.452751 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444199    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:52:40.452751 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444202    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:52:40.452751 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444204    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:52:40.452751 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444208    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:52:40.452751 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444212    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:52:40.452751 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444216    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:52:40.452751 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444219    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:52:40.452751 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444222    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:52:40.452751 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444224    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:52:40.452751 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444227    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:52:40.452751 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444230    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:52:40.452751 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444233    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:52:40.453537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444235    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:52:40.453537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444238    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:52:40.453537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444241    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:52:40.453537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444243    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:52:40.453537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444246    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:52:40.453537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444248    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:52:40.453537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444251    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:52:40.453537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444253    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:52:40.453537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444256    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:52:40.453537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444259    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:52:40.453537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444261    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:52:40.453537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444264    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:52:40.453537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444266    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:52:40.453537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444269    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:52:40.453537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444272    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:52:40.453537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444274    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:52:40.453537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444277    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:52:40.453537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444279    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:52:40.453537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444282    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:52:40.453537 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444285    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:52:40.454066 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444288    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:52:40.454066 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444291    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:52:40.454066 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444293    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:52:40.454066 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444296    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:52:40.454066 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444299    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:52:40.454066 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444302    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:52:40.454066 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444305    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:52:40.454066 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444308    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:52:40.454066 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444311    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:52:40.454066 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444313    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:52:40.454066 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444316    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:52:40.454066 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444319    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:52:40.454066 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444322    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:52:40.454066 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444324    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:52:40.454066 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444329    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:52:40.454066 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444332    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:52:40.454066 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444335    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:52:40.454066 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444338    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:52:40.454066 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444340    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:52:40.454066 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444343    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:52:40.454538 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444346    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:52:40.454538 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444348    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:52:40.454538 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444351    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:52:40.454538 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444353    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:52:40.454538 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.444379    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 07:52:40.454538 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444507    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:52:40.454538 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444511    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:52:40.454538 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444514    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:52:40.454538 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444517    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:52:40.454538 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444520    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:52:40.454538 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444522    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:52:40.454538 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444525    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:52:40.454538 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444528    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:52:40.454538 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444530    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:52:40.454538 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444533    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:52:40.454538 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444535    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:52:40.454932 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444538    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:52:40.454932 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444541    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:52:40.454932 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444544    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:52:40.454932 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444547    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:52:40.454932 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444550    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:52:40.454932 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444553    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:52:40.454932 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444555    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:52:40.454932 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444558    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:52:40.454932 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444561    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:52:40.454932 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444563    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:52:40.454932 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444566    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:52:40.454932 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444568    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:52:40.454932 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444571    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:52:40.454932 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444574    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:52:40.454932 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444576    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:52:40.454932 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444579    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:52:40.454932 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444581    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:52:40.454932 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444584    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:52:40.454932 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444587    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:52:40.454932 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444589    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:52:40.455386 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444592    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:52:40.455386 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444594    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:52:40.455386 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444597    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:52:40.455386 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444599    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:52:40.455386 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444602    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:52:40.455386 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444605    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:52:40.455386 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444607    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:52:40.455386 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444610    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:52:40.455386 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444612    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:52:40.455386 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444615    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:52:40.455386 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444618    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:52:40.455386 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444620    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:52:40.455386 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444623    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:52:40.455386 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444625    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:52:40.455386 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444628    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:52:40.455386 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444630    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:52:40.455386 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444633    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:52:40.455386 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444636    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:52:40.455386 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444639    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:52:40.455836 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444642    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:52:40.455836 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444644    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:52:40.455836 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444647    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:52:40.455836 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444649    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:52:40.455836 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444652    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:52:40.455836 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444655    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:52:40.455836 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444658    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:52:40.455836 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444660    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:52:40.455836 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444663    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:52:40.455836 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444666    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:52:40.455836 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444668    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:52:40.455836 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444671    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:52:40.455836 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444675    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:52:40.455836 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444678    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:52:40.455836 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444681    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:52:40.455836 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444684    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:52:40.455836 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444686    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:52:40.455836 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444689    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:52:40.455836 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444691    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:52:40.456336 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444694    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:52:40.456336 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444697    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:52:40.456336 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444699    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:52:40.456336 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444702    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:52:40.456336 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444704    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:52:40.456336 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444707    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:52:40.456336 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444709    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:52:40.456336 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444712    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:52:40.456336 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444715    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:52:40.456336 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444717    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:52:40.456336 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444720    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:52:40.456336 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444722    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:52:40.456336 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444725    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:52:40.456336 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444728    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:52:40.456336 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444731    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:52:40.456336 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444733    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:52:40.456336 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444736    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:52:40.456336 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444738    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:52:40.456336 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444741    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:52:40.456336 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444743    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:52:40.456842 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444746    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:52:40.456842 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444749    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:52:40.456842 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444751    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:52:40.456842 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444766    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:52:40.456842 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444769    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:52:40.456842 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444772    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:52:40.456842 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444776    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:52:40.456842 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444779    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:52:40.456842 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444782    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:52:40.456842 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444785    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:52:40.456842 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444788    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:52:40.456842 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444790    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:52:40.456842 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444793    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:52:40.456842 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444796    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:52:40.456842 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444798    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:52:40.456842 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444801    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:52:40.456842 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444803    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:52:40.457237 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.444826    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 07:52:40.457237 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444857    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:52:40.457237 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444861    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:52:40.457237 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444864    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:52:40.457237 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444866    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:52:40.457237 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444869    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:52:40.457237 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444872    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:52:40.457237 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444875    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:52:40.457237 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444878    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:52:40.457237 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444881    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:52:40.457237 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444883    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:52:40.457237 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444886    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:52:40.457237 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444889    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:52:40.457237 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444891    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:52:40.457237 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444894    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:52:40.457237 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444896    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:52:40.457650 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444899    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:52:40.457650 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444902    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:52:40.457650 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444904    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:52:40.457650 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444907    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:52:40.457650 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444909    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:52:40.457650 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444914    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:52:40.457650 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444917    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:52:40.457650 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444920    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:52:40.457650 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444922    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:52:40.457650 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444925    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:52:40.457650 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444928    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:52:40.457650 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444930    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:52:40.457650 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444935    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:52:40.457650 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444938    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:52:40.457650 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444940    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:52:40.457650 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444943    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:52:40.457650 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444945    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:52:40.457650 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444948    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:52:40.457650 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444950    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:52:40.458117 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444953    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:52:40.458117 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444956    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:52:40.458117 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444959    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:52:40.458117 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444961    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:52:40.458117 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444964    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:52:40.458117 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444966    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:52:40.458117 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444971    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:52:40.458117 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444975    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:52:40.458117 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444978    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:52:40.458117 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444982    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:52:40.458117 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444985    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:52:40.458117 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444988    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:52:40.458117 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444990    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:52:40.458117 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444993    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:52:40.458117 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444996    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:52:40.458117 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.444999    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:52:40.458117 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445002    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:52:40.458117 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445005    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:52:40.458117 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445008    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:52:40.458615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445011    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:52:40.458615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445013    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:52:40.458615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445016    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:52:40.458615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445018    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:52:40.458615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445021    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:52:40.458615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445024    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:52:40.458615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445028    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:52:40.458615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445030    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:52:40.458615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445033    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:52:40.458615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445035    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:52:40.458615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445038    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:52:40.458615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445041    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:52:40.458615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445044    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:52:40.458615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445046    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:52:40.458615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445049    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:52:40.458615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445052    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:52:40.458615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445055    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:52:40.458615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445057    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:52:40.458615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445060    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:52:40.458615 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445062    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:52:40.459205 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445066    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:52:40.459205 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445070    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:52:40.459205 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445073    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:52:40.459205 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445075    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:52:40.459205 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445078    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:52:40.459205 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445081    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:52:40.459205 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445083    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:52:40.459205 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445086    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:52:40.459205 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445088    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:52:40.459205 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445091    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:52:40.459205 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445093    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:52:40.459205 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445097    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:52:40.459205 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445100    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:52:40.459205 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445103    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:52:40.459205 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445105    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:52:40.459205 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445108    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:52:40.459205 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445111    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:52:40.459205 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445113    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:52:40.459205 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445118    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:52:40.459673 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445121    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:52:40.459673 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445123    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:52:40.459673 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445126    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:52:40.459673 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445129    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:52:40.459673 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445131    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:52:40.459673 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445134    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:52:40.459673 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445138    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:52:40.459673 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445140    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:52:40.459673 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445143    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:52:40.459673 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445145    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:52:40.459673 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445148    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:52:40.459673 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445151    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:52:40.459673 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445153    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:52:40.459673 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:40.445156    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:52:40.459673 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.445201    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 07:52:40.459673 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.446964    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 07:52:40.459673 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.447840    2579 server.go:1014] "Starting client certificate rotation"
Nov 03 07:52:40.459673 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.447936    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 07:52:40.459673 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.448576    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 07:52:40.460197 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.450617    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 07:52:40.460197 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.456189    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 07:52:40.468712 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.468627    2579 log.go:25] "Validated CRI v1 runtime API"
Nov 03 07:52:40.471524 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.471508    2579 log.go:25] "Validated CRI v1 image API"
Nov 03 07:52:40.472308 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.472284    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 07:52:40.474416 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.474391    2579 fs.go:135] Filesystem UUIDs: map[37e0c1fc-d3c8-4ff0-9a4d-34d6d14b78c3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d182e259-d93c-4f5d-bb2f-7384d6159fd3:/dev/nvme0n1p3]
Nov 03 07:52:40.474482 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.474415    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 07:52:40.475417 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.475398    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 07:52:40.479352 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.479230    2579 manager.go:217] Machine: {Timestamp:2026-11-03 07:52:40.477915395 +0000 UTC m=+0.345953107 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098515 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c30f8846b03117b3e346870769ecc SystemUUID:ec2c30f8-846b-0311-7b3e-346870769ecc BootID:0adaab61-516b-4804-89b7-57a4f7dc09e8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ce:d5:e5:a9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ce:d5:e5:a9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:10:87:b5:0a:55 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 07:52:40.479352 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.479344    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 07:52:40.479518 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.479471    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 07:52:40.483104 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.483079    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 07:52:40.483220 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.483104    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-123.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 07:52:40.483736 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.483725    2579 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 07:52:40.483778 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.483739    2579 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 07:52:40.483807 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.483778    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 07:52:40.483807 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.483790    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 07:52:40.484514 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.484504    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 07:52:40.488221 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.488209    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 07:52:40.488334 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.488325    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 07:52:40.488363 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.488354    2579 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 07:52:40.488389 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.488364    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 07:52:40.488389 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.488373    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 07:52:40.488389 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.488383    2579 kubelet.go:405] "Adding apiserver pod source"
Nov 03 07:52:40.488485 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.488390    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 07:52:40.488687 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.488670    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 07:52:40.492295 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.492269    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 07:52:40.497236 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.497220    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 07:52:40.498145 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.498131    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 07:52:40.498205 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.498161    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 07:52:40.499241 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.499229    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 07:52:40.499241 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.499245    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 07:52:40.499358 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.499251    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 07:52:40.499358 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.499257    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 07:52:40.499358 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.499263    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 07:52:40.499358 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.499268    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 07:52:40.499990 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.499981    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 07:52:40.499990 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.499990    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 07:52:40.500051 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.499997    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 07:52:40.500051 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.500003    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 07:52:40.500051 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.500011    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 07:52:40.500051 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.500020    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 07:52:40.501333 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.501323    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 07:52:40.501333 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.501333    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 07:52:40.502682 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:40.502656    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-123.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 07:52:40.502682 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:40.502664    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 07:52:40.504828 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.504816    2579 server.go:1263] "Started kubelet"
Nov 03 07:52:40.504966 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.504920    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 07:52:40.505034 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.504987    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 07:52:40.505270 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.504881    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 07:52:40.505622 ip-10-0-140-123 systemd[1]: Started Kubernetes Kubelet.
Nov 03 07:52:40.508137 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.508119    2579 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 07:52:40.508231 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.508124    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 07:52:40.510825 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.510807    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-123.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 07:52:40.511143 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:40.511128    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 07:52:40.511708 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:40.510703    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-123.us-east-2.compute.internal.18e479b2d038541e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-123.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-123.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 07:52:40.504792094 +0000 UTC m=+0.372829807,LastTimestamp:2026-11-03 07:52:40.504792094 +0000 UTC m=+0.372829807,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-123.us-east-2.compute.internal,}"
Nov 03 07:52:40.512808 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.512788    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 07:52:40.514124 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.514106    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 07:52:40.514254 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:40.514236    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-123.us-east-2.compute.internal\" not found"
Nov 03 07:52:40.514908 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.514889    2579 factory.go:55] Registering systemd factory
Nov 03 07:52:40.514908 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.514907    2579 factory.go:223] Registration of the systemd container factory successfully
Nov 03 07:52:40.515106 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.515093    2579 factory.go:153] Registering CRI-O factory
Nov 03 07:52:40.515106 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.515105    2579 factory.go:223] Registration of the crio container factory successfully
Nov 03 07:52:40.515207 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.515155    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 07:52:40.515207 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.515167    2579 factory.go:103] Registering Raw factory
Nov 03 07:52:40.515207 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.515178    2579 manager.go:1199] Started watching for new ooms in manager
Nov 03 07:52:40.515526 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.515506    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 07:52:40.515669 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.515653    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 07:52:40.515713 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.515673    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 07:52:40.515789 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.515779    2579 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 07:52:40.515789 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.515788    2579 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 07:52:40.516413 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.516384    2579 manager.go:319] Starting recovery of all containers
Nov 03 07:52:40.517426 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.517402    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b8xjt"
Nov 03 07:52:40.517534 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:40.517505    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-123.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 07:52:40.525258 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.525234    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b8xjt"
Nov 03 07:52:40.527325 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.527311    2579 manager.go:324] Recovery completed
Nov 03 07:52:40.529333 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:40.529312    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 07:52:40.531468 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.531455    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 07:52:40.533083 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.533066    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 07:52:40.533133 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.533102    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 07:52:40.533133 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.533117    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 07:52:40.533647 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.533634    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 07:52:40.533719 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.533635    2579 cpu_manager.go:226] "Starting" policy="none"
Nov 03 07:52:40.533777 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.533721    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 07:52:40.533820 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.533803    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 07:52:40.534277 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.534264    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 07:52:40.534338 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.534287    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 07:52:40.534338 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.534297    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 07:52:40.535814 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.535797    2579 policy_none.go:50] "Start"
Nov 03 07:52:40.535874 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.535819    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 07:52:40.535874 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.535832    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 07:52:40.538298 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.538282    2579 policy_none.go:44] "Start"
Nov 03 07:52:40.571330 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.571308    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 07:52:40.573659 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.573646    2579 manager.go:341] "Starting Device Plugin manager"
Nov 03 07:52:40.574313 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:40.574296    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 07:52:40.574313 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.574313    2579 server.go:87] "Starting device plugin registration server"
Nov 03 07:52:40.574597 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.574586    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 07:52:40.574652 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.574602    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 07:52:40.574699 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.574661    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 07:52:40.574785 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.574746    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 07:52:40.574785 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.574773    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 07:52:40.575193 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:40.575176    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 07:52:40.575275 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:40.575214    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-123.us-east-2.compute.internal\" not found"
Nov 03 07:52:40.675182 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.675165    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 07:52:40.676976 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.675965    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 07:52:40.676976 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.675994    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 07:52:40.676976 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.676004    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 07:52:40.676976 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.676027    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:52:40.680058 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.680003    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 07:52:40.681202 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.681188    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 07:52:40.681263 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.681209    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 07:52:40.681263 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.681228    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 07:52:40.681354 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:40.681263    2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 07:52:40.683909 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.683891    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:52:40.683978 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:40.683910    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-123.us-east-2.compute.internal\": node \"ip-10-0-140-123.us-east-2.compute.internal\" not found"
Nov 03 07:52:40.684988 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.684975    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 07:52:40.716889 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:40.716872    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-123.us-east-2.compute.internal\" not found"
Nov 03 07:52:40.782262 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.782229    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal"]
Nov 03 07:52:40.782328 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.782298    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 07:52:40.783037 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.783024    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 07:52:40.783098 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.783048    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 07:52:40.783098 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.783057    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 07:52:40.785103 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.785091    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 07:52:40.785250 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.785236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:52:40.785286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.785275    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 07:52:40.785732 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.785717    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 07:52:40.785732 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.785738    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 07:52:40.785865 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.785770    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 07:52:40.785865 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.785792    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 07:52:40.785865 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.785822    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 07:52:40.785865 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.785843    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 07:52:40.787833 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.787820    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:52:40.787886 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.787843    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 07:52:40.788504 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.788487    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 07:52:40.788580 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.788523    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 07:52:40.788580 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.788538    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 07:52:40.816978 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:40.816957    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-123.us-east-2.compute.internal\" not found"
Nov 03 07:52:40.818126 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:40.818108    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-123.us-east-2.compute.internal\" not found" node="ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:52:40.824383 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:40.824368    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-123.us-east-2.compute.internal\" not found" node="ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:52:40.917265 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:40.917240    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-123.us-east-2.compute.internal\" not found"
Nov 03 07:52:40.917348 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.917294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3012e3a7d17ffd2dd3688b2234e0123b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal\" (UID: \"3012e3a7d17ffd2dd3688b2234e0123b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:52:40.917348 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.917315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3012e3a7d17ffd2dd3688b2234e0123b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal\" (UID: \"3012e3a7d17ffd2dd3688b2234e0123b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:52:40.917348 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:40.917332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc5342d3d37d4243b5710bd6f9020da4-config\") pod \"kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal\" (UID: \"fc5342d3d37d4243b5710bd6f9020da4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:52:41.017946 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:41.017887    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-123.us-east-2.compute.internal\" not found"
Nov 03 07:52:41.017946 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:41.017908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc5342d3d37d4243b5710bd6f9020da4-config\") pod \"kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal\" (UID: \"fc5342d3d37d4243b5710bd6f9020da4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:52:41.017946 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:41.017942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3012e3a7d17ffd2dd3688b2234e0123b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal\" (UID: \"3012e3a7d17ffd2dd3688b2234e0123b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:52:41.018109 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:41.017967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3012e3a7d17ffd2dd3688b2234e0123b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal\" (UID: \"3012e3a7d17ffd2dd3688b2234e0123b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:52:41.018109 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:41.017988    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc5342d3d37d4243b5710bd6f9020da4-config\") pod \"kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal\" (UID: \"fc5342d3d37d4243b5710bd6f9020da4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:52:41.018109 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:41.018011    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3012e3a7d17ffd2dd3688b2234e0123b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal\" (UID: \"3012e3a7d17ffd2dd3688b2234e0123b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:52:41.018109 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:41.018018    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3012e3a7d17ffd2dd3688b2234e0123b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal\" (UID: \"3012e3a7d17ffd2dd3688b2234e0123b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:52:41.118276 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:41.118257    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-123.us-east-2.compute.internal\" not found"
Nov 03 07:52:41.121451 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:41.121436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:52:41.126947 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:41.126933    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:52:41.218486 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:41.218466    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-123.us-east-2.compute.internal\" not found"
Nov 03 07:52:41.318894 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:41.318851    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-123.us-east-2.compute.internal\" not found"
Nov 03 07:52:41.419177 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:41.419160    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-123.us-east-2.compute.internal\" not found"
Nov 03 07:52:41.448306 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:41.448292    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 07:52:41.448602 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:41.448396    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 07:52:41.520155 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:41.520128    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-123.us-east-2.compute.internal\" not found"
Nov 03 07:52:41.531343 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:41.531317    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 07:47:40 +0000 UTC" deadline="2028-08-08 12:36:19.851585 +0000 UTC"
Nov 03 07:52:41.531343 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:41.531342    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15460h43m38.320245936s"
Nov 03 07:52:41.565659 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:41.565637    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 07:52:41.614817 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:41.614779    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:52:41.618942 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:41.618927    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 07:52:41.623586 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:41.623572    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 07:52:41.625169 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:41.625155    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:52:41.634920 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:41.634901    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 07:52:41.744652 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:41.744516    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5342d3d37d4243b5710bd6f9020da4.slice/crio-1e702f08ed7ed20f697b4723e95f5b16c80b7b124dab15c88b47143f1709ef83 WatchSource:0}: Error finding container 1e702f08ed7ed20f697b4723e95f5b16c80b7b124dab15c88b47143f1709ef83: Status 404 returned error can't find the container with id 1e702f08ed7ed20f697b4723e95f5b16c80b7b124dab15c88b47143f1709ef83
Nov 03 07:52:41.744905 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:41.744886    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3012e3a7d17ffd2dd3688b2234e0123b.slice/crio-ad313963ad4d389ee2bd48bb11fa2d1f7b97e80459f0d44fc705931bce9194fc WatchSource:0}: Error finding container ad313963ad4d389ee2bd48bb11fa2d1f7b97e80459f0d44fc705931bce9194fc: Status 404 returned error can't find the container with id ad313963ad4d389ee2bd48bb11fa2d1f7b97e80459f0d44fc705931bce9194fc
Nov 03 07:52:41.749490 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:41.749478    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 07:52:42.036708 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.036643    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 07:52:42.465228 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.465159    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 07:52:42.490307 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.490289    2579 apiserver.go:52] "Watching apiserver"
Nov 03 07:52:42.498713 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.498691    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 07:52:42.498998 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.498982    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-ljbh2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db","openshift-dns/node-resolver-drdft","openshift-multus/multus-msq9p","openshift-network-operator/iptables-alerter-kj8jw","openshift-image-registry/node-ca-w9c2l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-h4dgx","kube-system/konnectivity-agent-9j2f9","kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-fgwf4","openshift-multus/network-metrics-daemon-x7cmt","openshift-network-diagnostics/network-check-target-c6bms","openshift-ovn-kubernetes/ovnkube-node-x6d47"]
Nov 03 07:52:42.501822 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.501801    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:52:42.501916 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:42.501881    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6bms" podUID="3a9e7292-b781-4a27-8a48-94f0a8843fc2"
Nov 03 07:52:42.503844 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.503827    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.505895 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.505876    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-drdft"
Nov 03 07:52:42.507569 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.507549    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bmmvq\""
Nov 03 07:52:42.507569 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.507562    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 07:52:42.507569 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.507553    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 07:52:42.507780 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.507654    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 07:52:42.508182 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.508167    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.508817 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.508801    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dc6md\""
Nov 03 07:52:42.508901 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.508865    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 07:52:42.508901 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.508865    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 07:52:42.510454 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.510435    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kj8jw"
Nov 03 07:52:42.510954 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.510940    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 07:52:42.511043 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.511026    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 07:52:42.511204 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.511192    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 07:52:42.511346 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.511327    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2mmmx\""
Nov 03 07:52:42.511579 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.511413    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 07:52:42.512655 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.512636    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w9c2l"
Nov 03 07:52:42.513162 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.513141    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 07:52:42.513256 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.513188    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 07:52:42.513308 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.513282    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkwf4\""
Nov 03 07:52:42.513431 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.513416    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 07:52:42.513528 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.513513    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 07:52:42.515010 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.514817    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:42.515010 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:42.514893    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljbh2" podUID="dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df"
Nov 03 07:52:42.515220 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.515206    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 07:52:42.515826 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.515660    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 07:52:42.515826 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.515697    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m4xzk\""
Nov 03 07:52:42.515826 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.515696    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 07:52:42.517368 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.517352    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.519813 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.519796    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9j2f9"
Nov 03 07:52:42.521587 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.521553    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.524649 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.524257    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.527266 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.527126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-device-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.527266 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.527181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-run-k8s-cni-cncf-io\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.527266 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.527214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-hostroot\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.527444 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.527315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2ed628f2-4323-47b6-ac03-b8567812a740-multus-daemon-config\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.527444 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.527402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q8m7\" (UniqueName: \"kubernetes.io/projected/bd5c9498-f849-4dd6-a9e7-27a0d5db4fde-kube-api-access-4q8m7\") pod \"iptables-alerter-kj8jw\" (UID: \"bd5c9498-f849-4dd6-a9e7-27a0d5db4fde\") " pod="openshift-network-operator/iptables-alerter-kj8jw"
Nov 03 07:52:42.530196 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.529159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtmpp\" (UniqueName: \"kubernetes.io/projected/5738916f-8202-40ed-823e-90f7c9cc4ec9-kube-api-access-rtmpp\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.530196 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.529187    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:52:42.530196 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.529213    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-multus-conf-dir\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.530196 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:42.529242    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7cmt" podUID="25a7353f-76ad-4ac0-92b8-186377723478"
Nov 03 07:52:42.530196 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.529259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd5c9498-f849-4dd6-a9e7-27a0d5db4fde-iptables-alerter-script\") pod \"iptables-alerter-kj8jw\" (UID: \"bd5c9498-f849-4dd6-a9e7-27a0d5db4fde\") " pod="openshift-network-operator/iptables-alerter-kj8jw"
Nov 03 07:52:42.530196 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.529289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f142ef3a-aa21-48eb-a403-8d51d843c72d-hosts-file\") pod \"node-resolver-drdft\" (UID: \"f142ef3a-aa21-48eb-a403-8d51d843c72d\") " pod="openshift-dns/node-resolver-drdft"
Nov 03 07:52:42.530196 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.529330    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-var-lib-cni-multus\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.530196 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.529374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k67j\" (UniqueName: \"kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j\") pod \"network-check-target-c6bms\" (UID: \"3a9e7292-b781-4a27-8a48-94f0a8843fc2\") " pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:52:42.530196 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.529410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-etc-selinux\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.530196 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.529508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-multus-socket-dir-parent\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.530196 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.529535    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-var-lib-cni-bin\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.530196 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.529561    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-kubelet-config\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:42.530196 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.529588    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19a8bde5-584f-4ff9-8e96-df33c5cda82d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.530196 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.529625    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-etc-kubernetes\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.530196 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.529651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19a8bde5-584f-4ff9-8e96-df33c5cda82d-system-cni-dir\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.530196 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.529718    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19a8bde5-584f-4ff9-8e96-df33c5cda82d-cnibin\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.530842 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.529827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-registration-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.530842 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.529870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-multus-cni-dir\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.530842 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.529909    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd5c9498-f849-4dd6-a9e7-27a0d5db4fde-host-slash\") pod \"iptables-alerter-kj8jw\" (UID: \"bd5c9498-f849-4dd6-a9e7-27a0d5db4fde\") " pod="openshift-network-operator/iptables-alerter-kj8jw"
Nov 03 07:52:42.530842 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.529937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67ff6855-c974-451d-b582-679e7f26e98d-serviceca\") pod \"node-ca-w9c2l\" (UID: \"67ff6855-c974-451d-b582-679e7f26e98d\") " pod="openshift-image-registry/node-ca-w9c2l"
Nov 03 07:52:42.530842 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.529963    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk9fs\" (UniqueName: \"kubernetes.io/projected/67ff6855-c974-451d-b582-679e7f26e98d-kube-api-access-xk9fs\") pod \"node-ca-w9c2l\" (UID: \"67ff6855-c974-451d-b582-679e7f26e98d\") " pod="openshift-image-registry/node-ca-w9c2l"
Nov 03 07:52:42.530842 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.530000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:42.530842 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.530068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4qdc\" (UniqueName: \"kubernetes.io/projected/19a8bde5-584f-4ff9-8e96-df33c5cda82d-kube-api-access-x4qdc\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.530842 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.530101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.530842 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.530131    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-run-netns\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.530842 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.530156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-var-lib-kubelet\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.530842 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.530185    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-run-multus-certs\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.530842 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.530708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67ff6855-c974-451d-b582-679e7f26e98d-host\") pod \"node-ca-w9c2l\" (UID: \"67ff6855-c974-451d-b582-679e7f26e98d\") " pod="openshift-image-registry/node-ca-w9c2l"
Nov 03 07:52:42.530842 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.530768    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19a8bde5-584f-4ff9-8e96-df33c5cda82d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.530842 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.530832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19a8bde5-584f-4ff9-8e96-df33c5cda82d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.531464 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.530874    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-cnibin\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.531464 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.530897    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ed628f2-4323-47b6-ac03-b8567812a740-cni-binary-copy\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.531464 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.530927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-dbus\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:42.531464 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.530944    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f142ef3a-aa21-48eb-a403-8d51d843c72d-tmp-dir\") pod \"node-resolver-drdft\" (UID: \"f142ef3a-aa21-48eb-a403-8d51d843c72d\") " pod="openshift-dns/node-resolver-drdft"
Nov 03 07:52:42.531464 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.530961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19a8bde5-584f-4ff9-8e96-df33c5cda82d-cni-binary-copy\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.531464 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.530985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-socket-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.531464 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.531038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-sys-fs\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.531464 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.531064    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-system-cni-dir\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.531464 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.531094    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv7pr\" (UniqueName: \"kubernetes.io/projected/2ed628f2-4323-47b6-ac03-b8567812a740-kube-api-access-gv7pr\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.531464 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.531120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lfcx\" (UniqueName: \"kubernetes.io/projected/f142ef3a-aa21-48eb-a403-8d51d843c72d-kube-api-access-8lfcx\") pod \"node-resolver-drdft\" (UID: \"f142ef3a-aa21-48eb-a403-8d51d843c72d\") " pod="openshift-dns/node-resolver-drdft"
Nov 03 07:52:42.540181 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.540165    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 07:52:42.540268 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.540207    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vh8ft"
Nov 03 07:52:42.545945 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.545928    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vh8ft"
Nov 03 07:52:42.599575 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.599557    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 07:52:42.599665 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.599575    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 07:52:42.599721 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.599680    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 07:52:42.599858 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.599824    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 07:52:42.599918 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.599884    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 07:52:42.599977 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.599957    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 07:52:42.600855 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.600838    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 07:52:42.600940 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.600882    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 07:52:42.600940 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.600914    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 07:52:42.600940 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.600918    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 07:52:42.601075 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.600880    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 07:52:42.601075 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.600979    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-thqm7\""
Nov 03 07:52:42.601075 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.600982    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k9glt\""
Nov 03 07:52:42.601522 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.601506    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fpwxk\""
Nov 03 07:52:42.601597 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.601502    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-c2t62\""
Nov 03 07:52:42.601646 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.601607    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 07:52:42.617047 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.617032    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 07:52:42.634391 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.631938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-device-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.634391 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.631976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2ed628f2-4323-47b6-ac03-b8567812a740-multus-daemon-config\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.634391 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632003    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/676416b5-105e-4650-baed-92baf169a9a6-konnectivity-ca\") pod \"konnectivity-agent-9j2f9\" (UID: \"676416b5-105e-4650-baed-92baf169a9a6\") " pod="kube-system/konnectivity-agent-9j2f9"
Nov 03 07:52:42.634391 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-log-socket\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.634391 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632047    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-cni-bin\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.634391 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtmpp\" (UniqueName: \"kubernetes.io/projected/5738916f-8202-40ed-823e-90f7c9cc4ec9-kube-api-access-rtmpp\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.634391 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-multus-conf-dir\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.634391 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd5c9498-f849-4dd6-a9e7-27a0d5db4fde-iptables-alerter-script\") pod \"iptables-alerter-kj8jw\" (UID: \"bd5c9498-f849-4dd6-a9e7-27a0d5db4fde\") " pod="openshift-network-operator/iptables-alerter-kj8jw"
Nov 03 07:52:42.634391 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-run-systemd\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.634391 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-run-openvswitch\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.634391 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632168    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-cni-netd\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.634391 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k67j\" (UniqueName: \"kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j\") pod \"network-check-target-c6bms\" (UID: \"3a9e7292-b781-4a27-8a48-94f0a8843fc2\") " pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:52:42.634391 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-etc-selinux\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.634391 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-multus-socket-dir-parent\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.634391 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-etc-kubernetes\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.634391 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632253    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/676416b5-105e-4650-baed-92baf169a9a6-agent-certs\") pod \"konnectivity-agent-9j2f9\" (UID: \"676416b5-105e-4650-baed-92baf169a9a6\") " pod="kube-system/konnectivity-agent-9j2f9"
Nov 03 07:52:42.634391 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632285    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79888519-e73a-4536-96e4-ce8e3ec99037-etc-tuned\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.635043 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-multus-cni-dir\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.635043 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67ff6855-c974-451d-b582-679e7f26e98d-serviceca\") pod \"node-ca-w9c2l\" (UID: \"67ff6855-c974-451d-b582-679e7f26e98d\") " pod="openshift-image-registry/node-ca-w9c2l"
Nov 03 07:52:42.635043 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632337    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-run-netns\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.635043 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.635043 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-run-multus-certs\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.635043 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-kubelet\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.635043 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-var-lib-openvswitch\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.635043 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632412    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/40259634-6b57-492e-8376-75ecf7900b06-env-overrides\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.635043 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-kubernetes\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.635043 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-host\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.635043 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7slwv\" (UniqueName: \"kubernetes.io/projected/25a7353f-76ad-4ac0-92b8-186377723478-kube-api-access-7slwv\") pod \"network-metrics-daemon-x7cmt\" (UID: \"25a7353f-76ad-4ac0-92b8-186377723478\") " pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:52:42.635043 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632493    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19a8bde5-584f-4ff9-8e96-df33c5cda82d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.635043 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f142ef3a-aa21-48eb-a403-8d51d843c72d-tmp-dir\") pod \"node-resolver-drdft\" (UID: \"f142ef3a-aa21-48eb-a403-8d51d843c72d\") " pod="openshift-dns/node-resolver-drdft"
Nov 03 07:52:42.635043 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632512    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2ed628f2-4323-47b6-ac03-b8567812a740-multus-daemon-config\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.635043 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/40259634-6b57-492e-8376-75ecf7900b06-ovn-node-metrics-cert\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.635043 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-sysctl-conf\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.635043 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632545    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-systemd\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.635489 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632605    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-device-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.635489 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-run\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.635489 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19a8bde5-584f-4ff9-8e96-df33c5cda82d-cni-binary-copy\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.635489 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-sys-fs\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.635489 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632705    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-system-cni-dir\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.635489 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv7pr\" (UniqueName: \"kubernetes.io/projected/2ed628f2-4323-47b6-ac03-b8567812a740-kube-api-access-gv7pr\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.635489 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632733    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lfcx\" (UniqueName: \"kubernetes.io/projected/f142ef3a-aa21-48eb-a403-8d51d843c72d-kube-api-access-8lfcx\") pod \"node-resolver-drdft\" (UID: \"f142ef3a-aa21-48eb-a403-8d51d843c72d\") " pod="openshift-dns/node-resolver-drdft"
Nov 03 07:52:42.635489 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-run-k8s-cni-cncf-io\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.635489 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632777    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-multus-conf-dir\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.635489 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632791    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-hostroot\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.635489 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.632821    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-hostroot\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.635489 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633128    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd5c9498-f849-4dd6-a9e7-27a0d5db4fde-iptables-alerter-script\") pod \"iptables-alerter-kj8jw\" (UID: \"bd5c9498-f849-4dd6-a9e7-27a0d5db4fde\") " pod="openshift-network-operator/iptables-alerter-kj8jw"
Nov 03 07:52:42.635489 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633156    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q8m7\" (UniqueName: \"kubernetes.io/projected/bd5c9498-f849-4dd6-a9e7-27a0d5db4fde-kube-api-access-4q8m7\") pod \"iptables-alerter-kj8jw\" (UID: \"bd5c9498-f849-4dd6-a9e7-27a0d5db4fde\") " pod="openshift-network-operator/iptables-alerter-kj8jw"
Nov 03 07:52:42.635489 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633184    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-modprobe-d\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.635489 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633202    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-sysconfig\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.635489 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633218    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52hng\" (UniqueName: \"kubernetes.io/projected/79888519-e73a-4536-96e4-ce8e3ec99037-kube-api-access-52hng\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.635489 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f142ef3a-aa21-48eb-a403-8d51d843c72d-hosts-file\") pod \"node-resolver-drdft\" (UID: \"f142ef3a-aa21-48eb-a403-8d51d843c72d\") " pod="openshift-dns/node-resolver-drdft"
Nov 03 07:52:42.636041 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633248    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-etc-openvswitch\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.636041 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633253    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19a8bde5-584f-4ff9-8e96-df33c5cda82d-cni-binary-copy\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.636041 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633268    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-node-log\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.636041 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633287    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-var-lib-kubelet\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.636041 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-var-lib-cni-multus\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.636041 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633317    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-sys-fs\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.636041 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-var-lib-cni-bin\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.636041 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-kubelet-config\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:42.636041 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633351    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.636041 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs\") pod \"network-metrics-daemon-x7cmt\" (UID: \"25a7353f-76ad-4ac0-92b8-186377723478\") " pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:52:42.636041 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19a8bde5-584f-4ff9-8e96-df33c5cda82d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.636041 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-slash\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.636041 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633411    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-run-ovn-kubernetes\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.636041 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633413    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-system-cni-dir\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.636041 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633493    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-etc-selinux\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.636041 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633591    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t57g\" (UniqueName: \"kubernetes.io/projected/40259634-6b57-492e-8376-75ecf7900b06-kube-api-access-9t57g\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.636580 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633638    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-multus-socket-dir-parent\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.636580 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633658    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-etc-kubernetes\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.636580 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633695    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-multus-cni-dir\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.636580 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633830    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f142ef3a-aa21-48eb-a403-8d51d843c72d-hosts-file\") pod \"node-resolver-drdft\" (UID: \"f142ef3a-aa21-48eb-a403-8d51d843c72d\") " pod="openshift-dns/node-resolver-drdft"
Nov 03 07:52:42.636580 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633850    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-kubelet-config\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:42.636580 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633896    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-var-lib-cni-bin\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.636580 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633904    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-run-k8s-cni-cncf-io\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.636580 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634049    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-run-multus-certs\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.636580 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634065    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19a8bde5-584f-4ff9-8e96-df33c5cda82d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.636580 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-lib-modules\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.636580 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634141    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79888519-e73a-4536-96e4-ce8e3ec99037-tmp\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.636580 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.633865    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-var-lib-cni-multus\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.636580 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634208    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.636580 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19a8bde5-584f-4ff9-8e96-df33c5cda82d-system-cni-dir\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.636580 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19a8bde5-584f-4ff9-8e96-df33c5cda82d-cnibin\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.636580 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634286    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19a8bde5-584f-4ff9-8e96-df33c5cda82d-system-cni-dir\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.636580 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634331    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f142ef3a-aa21-48eb-a403-8d51d843c72d-tmp-dir\") pod \"node-resolver-drdft\" (UID: \"f142ef3a-aa21-48eb-a403-8d51d843c72d\") " pod="openshift-dns/node-resolver-drdft"
Nov 03 07:52:42.636580 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634336    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67ff6855-c974-451d-b582-679e7f26e98d-serviceca\") pod \"node-ca-w9c2l\" (UID: \"67ff6855-c974-451d-b582-679e7f26e98d\") " pod="openshift-image-registry/node-ca-w9c2l"
Nov 03 07:52:42.637377 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-registration-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.637377 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634410    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19a8bde5-584f-4ff9-8e96-df33c5cda82d-cnibin\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.637377 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd5c9498-f849-4dd6-a9e7-27a0d5db4fde-host-slash\") pod \"iptables-alerter-kj8jw\" (UID: \"bd5c9498-f849-4dd6-a9e7-27a0d5db4fde\") " pod="openshift-network-operator/iptables-alerter-kj8jw"
Nov 03 07:52:42.637377 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634430    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19a8bde5-584f-4ff9-8e96-df33c5cda82d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.637377 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634442    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd5c9498-f849-4dd6-a9e7-27a0d5db4fde-host-slash\") pod \"iptables-alerter-kj8jw\" (UID: \"bd5c9498-f849-4dd6-a9e7-27a0d5db4fde\") " pod="openshift-network-operator/iptables-alerter-kj8jw"
Nov 03 07:52:42.637377 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634456    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk9fs\" (UniqueName: \"kubernetes.io/projected/67ff6855-c974-451d-b582-679e7f26e98d-kube-api-access-xk9fs\") pod \"node-ca-w9c2l\" (UID: \"67ff6855-c974-451d-b582-679e7f26e98d\") " pod="openshift-image-registry/node-ca-w9c2l"
Nov 03 07:52:42.637377 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634459    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-registration-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.637377 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634491    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:42.637377 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4qdc\" (UniqueName: \"kubernetes.io/projected/19a8bde5-584f-4ff9-8e96-df33c5cda82d-kube-api-access-x4qdc\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.637377 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:42.634550    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 07:52:42.637377 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:42.634676    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret podName:dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df nodeName:}" failed. No retries permitted until 2026-11-03 07:52:43.134656766 +0000 UTC m=+3.002694480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret") pod "global-pull-secret-syncer-ljbh2" (UID: "dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df") : object "kube-system"/"original-pull-secret" not registered
Nov 03 07:52:42.637377 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-run-netns\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.637377 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-var-lib-kubelet\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.637377 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67ff6855-c974-451d-b582-679e7f26e98d-host\") pod \"node-ca-w9c2l\" (UID: \"67ff6855-c974-451d-b582-679e7f26e98d\") " pod="openshift-image-registry/node-ca-w9c2l"
Nov 03 07:52:42.637377 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634784    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-run-netns\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.637377 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-run-ovn\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.637377 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634826    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-var-lib-kubelet\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.637994 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634852    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/40259634-6b57-492e-8376-75ecf7900b06-ovnkube-script-lib\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.637994 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634890    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67ff6855-c974-451d-b582-679e7f26e98d-host\") pod \"node-ca-w9c2l\" (UID: \"67ff6855-c974-451d-b582-679e7f26e98d\") " pod="openshift-image-registry/node-ca-w9c2l"
Nov 03 07:52:42.637994 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634917    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-sys\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.637994 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19a8bde5-584f-4ff9-8e96-df33c5cda82d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.637994 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.634990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-cnibin\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.637994 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.635042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ed628f2-4323-47b6-ac03-b8567812a740-cni-binary-copy\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.637994 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.635075    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-dbus\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:42.637994 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.635094    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-cnibin\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.637994 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.635141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-socket-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.637994 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.635168    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-dbus\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:42.637994 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.635187    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-systemd-units\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.637994 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.635220    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/40259634-6b57-492e-8376-75ecf7900b06-ovnkube-config\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.637994 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.635238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-sysctl-d\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.637994 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.635311    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-socket-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.637994 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.635801    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19a8bde5-584f-4ff9-8e96-df33c5cda82d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.637994 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.635958    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ed628f2-4323-47b6-ac03-b8567812a740-cni-binary-copy\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.642519 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.642495    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 07:52:42.649287 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:42.648043    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 07:52:42.649287 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:42.648069    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 07:52:42.649287 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:42.648088    2579 projected.go:196] Error preparing data for projected volume kube-api-access-7k67j for pod openshift-network-diagnostics/network-check-target-c6bms: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:52:42.649287 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:42.648176    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j podName:3a9e7292-b781-4a27-8a48-94f0a8843fc2 nodeName:}" failed. No retries permitted until 2026-11-03 07:52:43.148122953 +0000 UTC m=+3.016160673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7k67j" (UniqueName: "kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j") pod "network-check-target-c6bms" (UID: "3a9e7292-b781-4a27-8a48-94f0a8843fc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:52:42.649287 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.648389    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtmpp\" (UniqueName: \"kubernetes.io/projected/5738916f-8202-40ed-823e-90f7c9cc4ec9-kube-api-access-rtmpp\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.652801 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.651993    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk9fs\" (UniqueName: \"kubernetes.io/projected/67ff6855-c974-451d-b582-679e7f26e98d-kube-api-access-xk9fs\") pod \"node-ca-w9c2l\" (UID: \"67ff6855-c974-451d-b582-679e7f26e98d\") " pod="openshift-image-registry/node-ca-w9c2l"
Nov 03 07:52:42.652801 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.652004    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4qdc\" (UniqueName: \"kubernetes.io/projected/19a8bde5-584f-4ff9-8e96-df33c5cda82d-kube-api-access-x4qdc\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.652801 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.652015    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lfcx\" (UniqueName: \"kubernetes.io/projected/f142ef3a-aa21-48eb-a403-8d51d843c72d-kube-api-access-8lfcx\") pod \"node-resolver-drdft\" (UID: \"f142ef3a-aa21-48eb-a403-8d51d843c72d\") " pod="openshift-dns/node-resolver-drdft"
Nov 03 07:52:42.652801 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.652351    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q8m7\" (UniqueName: \"kubernetes.io/projected/bd5c9498-f849-4dd6-a9e7-27a0d5db4fde-kube-api-access-4q8m7\") pod \"iptables-alerter-kj8jw\" (UID: \"bd5c9498-f849-4dd6-a9e7-27a0d5db4fde\") " pod="openshift-network-operator/iptables-alerter-kj8jw"
Nov 03 07:52:42.653022 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.652999    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv7pr\" (UniqueName: \"kubernetes.io/projected/2ed628f2-4323-47b6-ac03-b8567812a740-kube-api-access-gv7pr\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.685601 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.685561    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal" event={"ID":"3012e3a7d17ffd2dd3688b2234e0123b","Type":"ContainerStarted","Data":"ad313963ad4d389ee2bd48bb11fa2d1f7b97e80459f0d44fc705931bce9194fc"}
Nov 03 07:52:42.687095 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.687072    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal" event={"ID":"fc5342d3d37d4243b5710bd6f9020da4","Type":"ContainerStarted","Data":"1e702f08ed7ed20f697b4723e95f5b16c80b7b124dab15c88b47143f1709ef83"}
Nov 03 07:52:42.737737 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.736539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-systemd-units\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.737737 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.736574    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/40259634-6b57-492e-8376-75ecf7900b06-ovnkube-config\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.737737 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.736597    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-sysctl-d\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.737737 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.736618    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/676416b5-105e-4650-baed-92baf169a9a6-konnectivity-ca\") pod \"konnectivity-agent-9j2f9\" (UID: \"676416b5-105e-4650-baed-92baf169a9a6\") " pod="kube-system/konnectivity-agent-9j2f9"
Nov 03 07:52:42.737737 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.736631    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-log-socket\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.737737 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.736644    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-cni-bin\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.737737 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.736658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-run-systemd\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.737737 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.736672    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-run-openvswitch\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.737737 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.736689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-cni-netd\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.737737 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.736732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/676416b5-105e-4650-baed-92baf169a9a6-agent-certs\") pod \"konnectivity-agent-9j2f9\" (UID: \"676416b5-105e-4650-baed-92baf169a9a6\") " pod="kube-system/konnectivity-agent-9j2f9"
Nov 03 07:52:42.737737 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.736815    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-log-socket\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.737737 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.736816    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-run-openvswitch\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.737737 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.736846    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-cni-bin\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.737737 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.736862    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-systemd-units\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.737737 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.736888    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-run-systemd\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.737737 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.736920    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-cni-netd\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.737737 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.737032    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-sysctl-d\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.737737 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.737362    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/40259634-6b57-492e-8376-75ecf7900b06-ovnkube-config\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.738557 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.737445    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/676416b5-105e-4650-baed-92baf169a9a6-konnectivity-ca\") pod \"konnectivity-agent-9j2f9\" (UID: \"676416b5-105e-4650-baed-92baf169a9a6\") " pod="kube-system/konnectivity-agent-9j2f9"
Nov 03 07:52:42.738621 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.738603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79888519-e73a-4536-96e4-ce8e3ec99037-etc-tuned\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.738691 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.738647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-run-netns\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.738691 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.738682    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-kubelet\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.738813 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.738713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-var-lib-openvswitch\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.738813 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.738734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/40259634-6b57-492e-8376-75ecf7900b06-env-overrides\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.738813 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.738776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-kubernetes\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.738813 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.738796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-host\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.738984 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.738833    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7slwv\" (UniqueName: \"kubernetes.io/projected/25a7353f-76ad-4ac0-92b8-186377723478-kube-api-access-7slwv\") pod \"network-metrics-daemon-x7cmt\" (UID: \"25a7353f-76ad-4ac0-92b8-186377723478\") " pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:52:42.738984 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.738857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/40259634-6b57-492e-8376-75ecf7900b06-ovn-node-metrics-cert\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.738984 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.738878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-sysctl-conf\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.738984 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.738898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-systemd\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.738984 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.738918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-run\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.738984 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.738947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-modprobe-d\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.738984 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.738969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-sysconfig\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.739271 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.738995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52hng\" (UniqueName: \"kubernetes.io/projected/79888519-e73a-4536-96e4-ce8e3ec99037-kube-api-access-52hng\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.739271 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-etc-openvswitch\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.739271 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-node-log\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.739271 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-var-lib-kubelet\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.739271 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739095    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.739271 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs\") pod \"network-metrics-daemon-x7cmt\" (UID: \"25a7353f-76ad-4ac0-92b8-186377723478\") " pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:52:42.739271 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-slash\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.739271 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-run-ovn-kubernetes\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.739271 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739192    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t57g\" (UniqueName: \"kubernetes.io/projected/40259634-6b57-492e-8376-75ecf7900b06-kube-api-access-9t57g\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.739271 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-lib-modules\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.739271 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739234    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79888519-e73a-4536-96e4-ce8e3ec99037-tmp\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.739769 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-run-ovn\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.739769 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739314    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/40259634-6b57-492e-8376-75ecf7900b06-ovnkube-script-lib\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.739769 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-sys\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.739769 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739419    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-sys\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.739769 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739459    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-run-netns\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.739769 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739493    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-kubelet\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.739769 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739526    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-var-lib-openvswitch\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.739769 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739582    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-etc-openvswitch\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.739769 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739627    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-kubernetes\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.739769 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739653    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-host\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.740191 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739880    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/40259634-6b57-492e-8376-75ecf7900b06-env-overrides\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.740191 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739937    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-node-log\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.740191 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739952    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79888519-e73a-4536-96e4-ce8e3ec99037-etc-tuned\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.740191 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.739987    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-var-lib-kubelet\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.740191 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.740044    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.740191 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.740068    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-run\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.740191 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.740115    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-systemd\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.740191 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:42.740120    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:52:42.740191 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.740165    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-sysconfig\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.740191 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:42.740182    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs podName:25a7353f-76ad-4ac0-92b8-186377723478 nodeName:}" failed. No retries permitted until 2026-11-03 07:52:43.24016826 +0000 UTC m=+3.108205974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs") pod "network-metrics-daemon-x7cmt" (UID: "25a7353f-76ad-4ac0-92b8-186377723478") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:52:42.740641 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.740224    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-modprobe-d\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.740641 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.740233    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-slash\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.740641 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.740276    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-run-ovn-kubernetes\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.740641 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.740415    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-run-ovn\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.740641 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.740487    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-lib-modules\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.740641 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.740560    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-sysctl-conf\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.741167 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.741143    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/40259634-6b57-492e-8376-75ecf7900b06-ovnkube-script-lib\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.741656 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.741638    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/40259634-6b57-492e-8376-75ecf7900b06-ovn-node-metrics-cert\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.742233 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.742210    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/676416b5-105e-4650-baed-92baf169a9a6-agent-certs\") pod \"konnectivity-agent-9j2f9\" (UID: \"676416b5-105e-4650-baed-92baf169a9a6\") " pod="kube-system/konnectivity-agent-9j2f9"
Nov 03 07:52:42.743745 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.743717    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79888519-e73a-4536-96e4-ce8e3ec99037-tmp\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.748701 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.748651    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7slwv\" (UniqueName: \"kubernetes.io/projected/25a7353f-76ad-4ac0-92b8-186377723478-kube-api-access-7slwv\") pod \"network-metrics-daemon-x7cmt\" (UID: \"25a7353f-76ad-4ac0-92b8-186377723478\") " pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:52:42.748827 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.748811    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t57g\" (UniqueName: \"kubernetes.io/projected/40259634-6b57-492e-8376-75ecf7900b06-kube-api-access-9t57g\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.748903 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.748884    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52hng\" (UniqueName: \"kubernetes.io/projected/79888519-e73a-4536-96e4-ce8e3ec99037-kube-api-access-52hng\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.821031 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.821004    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:52:42.824302 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.824282    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-drdft"
Nov 03 07:52:42.831534 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:42.831507    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5738916f_8202_40ed_823e_90f7c9cc4ec9.slice/crio-33f0b316861516f8b3fea6f55511942d9d0c997927043f4ea9ed2846ec122015 WatchSource:0}: Error finding container 33f0b316861516f8b3fea6f55511942d9d0c997927043f4ea9ed2846ec122015: Status 404 returned error can't find the container with id 33f0b316861516f8b3fea6f55511942d9d0c997927043f4ea9ed2846ec122015
Nov 03 07:52:42.831605 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.831584    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-msq9p"
Nov 03 07:52:42.838730 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.838708    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kj8jw"
Nov 03 07:52:42.840443 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:42.840421    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed628f2_4323_47b6_ac03_b8567812a740.slice/crio-aa397f6e4634b8e2a019fa48addbf573d101c57973553024e7e833c9a2207d08 WatchSource:0}: Error finding container aa397f6e4634b8e2a019fa48addbf573d101c57973553024e7e833c9a2207d08: Status 404 returned error can't find the container with id aa397f6e4634b8e2a019fa48addbf573d101c57973553024e7e833c9a2207d08
Nov 03 07:52:42.845991 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.845970    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w9c2l"
Nov 03 07:52:42.847463 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:42.847442    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd5c9498_f849_4dd6_a9e7_27a0d5db4fde.slice/crio-424d19eae090d708343b986bdafa6c2807fa7980e6a4e15c660c8513e9dc243b WatchSource:0}: Error finding container 424d19eae090d708343b986bdafa6c2807fa7980e6a4e15c660c8513e9dc243b: Status 404 returned error can't find the container with id 424d19eae090d708343b986bdafa6c2807fa7980e6a4e15c660c8513e9dc243b
Nov 03 07:52:42.854857 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:42.854832    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ff6855_c974_451d_b582_679e7f26e98d.slice/crio-3db1086282f68b516f21655c635c9ae9a15e019a958eeb27efacbb40a9e3bf53 WatchSource:0}: Error finding container 3db1086282f68b516f21655c635c9ae9a15e019a958eeb27efacbb40a9e3bf53: Status 404 returned error can't find the container with id 3db1086282f68b516f21655c635c9ae9a15e019a958eeb27efacbb40a9e3bf53
Nov 03 07:52:42.866362 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.866342    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9j2f9"
Nov 03 07:52:42.868419 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.868396    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:52:42.878021 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:42.878003    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676416b5_105e_4650_baed_92baf169a9a6.slice/crio-ac82ecdb883da07b23dfd96b1c3cb432ae1ea597e9e36a08d7d32f83d64d3486 WatchSource:0}: Error finding container ac82ecdb883da07b23dfd96b1c3cb432ae1ea597e9e36a08d7d32f83d64d3486: Status 404 returned error can't find the container with id ac82ecdb883da07b23dfd96b1c3cb432ae1ea597e9e36a08d7d32f83d64d3486
Nov 03 07:52:42.878525 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.878508    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:52:42.881176 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:42.881157    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a8bde5_584f_4ff9_8e96_df33c5cda82d.slice/crio-910927b265f7362ee19a8aa7142206e182ead12e4de172025dc36d0ff37c70cb WatchSource:0}: Error finding container 910927b265f7362ee19a8aa7142206e182ead12e4de172025dc36d0ff37c70cb: Status 404 returned error can't find the container with id 910927b265f7362ee19a8aa7142206e182ead12e4de172025dc36d0ff37c70cb
Nov 03 07:52:42.886389 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:42.886372    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:52:42.887258 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:42.887242    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79888519_e73a_4536_96e4_ce8e3ec99037.slice/crio-208339fedfd203ca0612a484883dcee5846a4a886da948f4284ea71101d2afba WatchSource:0}: Error finding container 208339fedfd203ca0612a484883dcee5846a4a886da948f4284ea71101d2afba: Status 404 returned error can't find the container with id 208339fedfd203ca0612a484883dcee5846a4a886da948f4284ea71101d2afba
Nov 03 07:52:42.895853 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:52:42.895834    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40259634_6b57_492e_8376_75ecf7900b06.slice/crio-f71afc8fd4847c8d181bbb47535193beca676f165cb4f4aa4db71111413e4f0d WatchSource:0}: Error finding container f71afc8fd4847c8d181bbb47535193beca676f165cb4f4aa4db71111413e4f0d: Status 404 returned error can't find the container with id f71afc8fd4847c8d181bbb47535193beca676f165cb4f4aa4db71111413e4f0d
Nov 03 07:52:43.142525 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:43.142352    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:43.142525 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:43.142504    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 07:52:43.142709 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:43.142574    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret podName:dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df nodeName:}" failed. No retries permitted until 2026-11-03 07:52:44.142555859 +0000 UTC m=+4.010593557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret") pod "global-pull-secret-syncer-ljbh2" (UID: "dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df") : object "kube-system"/"original-pull-secret" not registered
Nov 03 07:52:43.243030 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:43.243010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k67j\" (UniqueName: \"kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j\") pod \"network-check-target-c6bms\" (UID: \"3a9e7292-b781-4a27-8a48-94f0a8843fc2\") " pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:52:43.243077 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:43.243049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs\") pod \"network-metrics-daemon-x7cmt\" (UID: \"25a7353f-76ad-4ac0-92b8-186377723478\") " pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:52:43.243148 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:43.243138    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:52:43.243175 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:43.243150    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 07:52:43.243175 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:43.243167    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 07:52:43.243175 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:43.243175    2579 projected.go:196] Error preparing data for projected volume kube-api-access-7k67j for pod openshift-network-diagnostics/network-check-target-c6bms: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:52:43.243255 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:43.243180    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs podName:25a7353f-76ad-4ac0-92b8-186377723478 nodeName:}" failed. No retries permitted until 2026-11-03 07:52:44.243169658 +0000 UTC m=+4.111207356 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs") pod "network-metrics-daemon-x7cmt" (UID: "25a7353f-76ad-4ac0-92b8-186377723478") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:52:43.243255 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:43.243210    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j podName:3a9e7292-b781-4a27-8a48-94f0a8843fc2 nodeName:}" failed. No retries permitted until 2026-11-03 07:52:44.243199782 +0000 UTC m=+4.111237491 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7k67j" (UniqueName: "kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j") pod "network-check-target-c6bms" (UID: "3a9e7292-b781-4a27-8a48-94f0a8843fc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:52:43.548631 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:43.548522    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 07:47:42 +0000 UTC" deadline="2028-05-23 12:12:50.434945984 +0000 UTC"
Nov 03 07:52:43.548631 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:43.548583    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13612h20m6.886367082s"
Nov 03 07:52:43.681453 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:43.681419    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:52:43.681652 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:43.681556    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7cmt" podUID="25a7353f-76ad-4ac0-92b8-186377723478"
Nov 03 07:52:43.696144 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:43.696062    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-msq9p" event={"ID":"2ed628f2-4323-47b6-ac03-b8567812a740","Type":"ContainerStarted","Data":"aa397f6e4634b8e2a019fa48addbf573d101c57973553024e7e833c9a2207d08"}
Nov 03 07:52:43.714449 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:43.714416    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db" event={"ID":"5738916f-8202-40ed-823e-90f7c9cc4ec9","Type":"ContainerStarted","Data":"33f0b316861516f8b3fea6f55511942d9d0c997927043f4ea9ed2846ec122015"}
Nov 03 07:52:43.717372 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:43.717339    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal" event={"ID":"fc5342d3d37d4243b5710bd6f9020da4","Type":"ContainerStarted","Data":"ae5476d431be031cbd30539a8848a33f604db2815ec69d0c5a52b42d245db564"}
Nov 03 07:52:43.720792 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:43.720684    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6d47" event={"ID":"40259634-6b57-492e-8376-75ecf7900b06","Type":"ContainerStarted","Data":"f71afc8fd4847c8d181bbb47535193beca676f165cb4f4aa4db71111413e4f0d"}
Nov 03 07:52:43.728256 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:43.724622    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4dgx" event={"ID":"19a8bde5-584f-4ff9-8e96-df33c5cda82d","Type":"ContainerStarted","Data":"910927b265f7362ee19a8aa7142206e182ead12e4de172025dc36d0ff37c70cb"}
Nov 03 07:52:43.733642 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:43.733611    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w9c2l" event={"ID":"67ff6855-c974-451d-b582-679e7f26e98d","Type":"ContainerStarted","Data":"3db1086282f68b516f21655c635c9ae9a15e019a958eeb27efacbb40a9e3bf53"}
Nov 03 07:52:43.747316 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:43.747261    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kj8jw" event={"ID":"bd5c9498-f849-4dd6-a9e7-27a0d5db4fde","Type":"ContainerStarted","Data":"424d19eae090d708343b986bdafa6c2807fa7980e6a4e15c660c8513e9dc243b"}
Nov 03 07:52:43.762686 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:43.762634    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-drdft" event={"ID":"f142ef3a-aa21-48eb-a403-8d51d843c72d","Type":"ContainerStarted","Data":"7378c45cd11e438cb3bb6d12e79041ef787a8b5093548efa0943413db3b2d294"}
Nov 03 07:52:43.776803 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:43.776361    2579 generic.go:356] "Generic (PLEG): container finished" podID="3012e3a7d17ffd2dd3688b2234e0123b" containerID="99692de9354655f075332cd7bb12593f7b6658aea489d064d59ce3c6cd4ffcd2" exitCode=0
Nov 03 07:52:43.776803 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:43.776431    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal" event={"ID":"3012e3a7d17ffd2dd3688b2234e0123b","Type":"ContainerDied","Data":"99692de9354655f075332cd7bb12593f7b6658aea489d064d59ce3c6cd4ffcd2"}
Nov 03 07:52:43.793491 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:43.793277    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal" podStartSLOduration=2.793261526 podStartE2EDuration="2.793261526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:52:43.733034339 +0000 UTC m=+3.601072071" watchObservedRunningTime="2026-11-03 07:52:43.793261526 +0000 UTC m=+3.661299248"
Nov 03 07:52:43.799978 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:43.799313    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fgwf4" event={"ID":"79888519-e73a-4536-96e4-ce8e3ec99037","Type":"ContainerStarted","Data":"208339fedfd203ca0612a484883dcee5846a4a886da948f4284ea71101d2afba"}
Nov 03 07:52:43.811774 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:43.811691    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9j2f9" event={"ID":"676416b5-105e-4650-baed-92baf169a9a6","Type":"ContainerStarted","Data":"ac82ecdb883da07b23dfd96b1c3cb432ae1ea597e9e36a08d7d32f83d64d3486"}
Nov 03 07:52:44.152220 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:44.151677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:44.152220 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:44.151834    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 07:52:44.152220 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:44.151892    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret podName:dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df nodeName:}" failed. No retries permitted until 2026-11-03 07:52:46.151874984 +0000 UTC m=+6.019912697 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret") pod "global-pull-secret-syncer-ljbh2" (UID: "dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df") : object "kube-system"/"original-pull-secret" not registered
Nov 03 07:52:44.252914 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:44.252145    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k67j\" (UniqueName: \"kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j\") pod \"network-check-target-c6bms\" (UID: \"3a9e7292-b781-4a27-8a48-94f0a8843fc2\") " pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:52:44.252914 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:44.252252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs\") pod \"network-metrics-daemon-x7cmt\" (UID: \"25a7353f-76ad-4ac0-92b8-186377723478\") " pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:52:44.252914 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:44.252385    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:52:44.252914 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:44.252442    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs podName:25a7353f-76ad-4ac0-92b8-186377723478 nodeName:}" failed. No retries permitted until 2026-11-03 07:52:46.252425397 +0000 UTC m=+6.120463110 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs") pod "network-metrics-daemon-x7cmt" (UID: "25a7353f-76ad-4ac0-92b8-186377723478") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:52:44.252914 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:44.252523    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 07:52:44.252914 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:44.252537    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 07:52:44.252914 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:44.252550    2579 projected.go:196] Error preparing data for projected volume kube-api-access-7k67j for pod openshift-network-diagnostics/network-check-target-c6bms: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:52:44.252914 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:44.252584    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j podName:3a9e7292-b781-4a27-8a48-94f0a8843fc2 nodeName:}" failed. No retries permitted until 2026-11-03 07:52:46.252572063 +0000 UTC m=+6.120609775 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7k67j" (UniqueName: "kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j") pod "network-check-target-c6bms" (UID: "3a9e7292-b781-4a27-8a48-94f0a8843fc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:52:44.549697 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:44.549595    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 07:47:42 +0000 UTC" deadline="2028-04-10 19:57:36.188890263 +0000 UTC"
Nov 03 07:52:44.549697 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:44.549654    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12588h4m51.639239844s"
Nov 03 07:52:44.684342 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:44.683892    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:52:44.684342 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:44.684023    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6bms" podUID="3a9e7292-b781-4a27-8a48-94f0a8843fc2"
Nov 03 07:52:44.684629 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:44.684482    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:44.684629 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:44.684577    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljbh2" podUID="dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df"
Nov 03 07:52:44.831193 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:44.830659    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal" event={"ID":"3012e3a7d17ffd2dd3688b2234e0123b","Type":"ContainerStarted","Data":"48146d79563f9d6c4482160a29a06f7f0badfa26caf9e8c2d3c108bb5fad1c60"}
Nov 03 07:52:45.682145 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:45.682108    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:52:45.682528 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:45.682236    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7cmt" podUID="25a7353f-76ad-4ac0-92b8-186377723478"
Nov 03 07:52:45.837795 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:45.837631    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4dgx" event={"ID":"19a8bde5-584f-4ff9-8e96-df33c5cda82d","Type":"ContainerStarted","Data":"fd7003fa158b4562f41e538a4a9a79512b3747e95a33da3c6e26e2a9f5b4966a"}
Nov 03 07:52:45.860842 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:45.860424    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal" podStartSLOduration=4.8604084610000005 podStartE2EDuration="4.860408461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:52:44.846503323 +0000 UTC m=+4.714541046" watchObservedRunningTime="2026-11-03 07:52:45.860408461 +0000 UTC m=+5.728446181"
Nov 03 07:52:46.169498 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:46.169408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:46.169649 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:46.169570    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 07:52:46.169649 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:46.169630    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret podName:dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df nodeName:}" failed. No retries permitted until 2026-11-03 07:52:50.169613014 +0000 UTC m=+10.037650718 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret") pod "global-pull-secret-syncer-ljbh2" (UID: "dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df") : object "kube-system"/"original-pull-secret" not registered
Nov 03 07:52:46.270633 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:46.270357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k67j\" (UniqueName: \"kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j\") pod \"network-check-target-c6bms\" (UID: \"3a9e7292-b781-4a27-8a48-94f0a8843fc2\") " pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:52:46.270633 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:46.270431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs\") pod \"network-metrics-daemon-x7cmt\" (UID: \"25a7353f-76ad-4ac0-92b8-186377723478\") " pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:52:46.270633 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:46.270507    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 07:52:46.270633 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:46.270530    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 07:52:46.270633 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:46.270539    2579 projected.go:196] Error preparing data for projected volume kube-api-access-7k67j for pod openshift-network-diagnostics/network-check-target-c6bms: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:52:46.270633 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:46.270558    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:52:46.270633 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:46.270584    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j podName:3a9e7292-b781-4a27-8a48-94f0a8843fc2 nodeName:}" failed. No retries permitted until 2026-11-03 07:52:50.270568953 +0000 UTC m=+10.138606653 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7k67j" (UniqueName: "kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j") pod "network-check-target-c6bms" (UID: "3a9e7292-b781-4a27-8a48-94f0a8843fc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:52:46.270633 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:46.270608    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs podName:25a7353f-76ad-4ac0-92b8-186377723478 nodeName:}" failed. No retries permitted until 2026-11-03 07:52:50.270592956 +0000 UTC m=+10.138630660 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs") pod "network-metrics-daemon-x7cmt" (UID: "25a7353f-76ad-4ac0-92b8-186377723478") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:52:46.683771 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:46.683722    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:52:46.684222 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:46.683867    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6bms" podUID="3a9e7292-b781-4a27-8a48-94f0a8843fc2"
Nov 03 07:52:46.684279 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:46.684248    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:46.684367 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:46.684343    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljbh2" podUID="dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df"
Nov 03 07:52:46.843478 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:46.842722    2579 generic.go:356] "Generic (PLEG): container finished" podID="19a8bde5-584f-4ff9-8e96-df33c5cda82d" containerID="fd7003fa158b4562f41e538a4a9a79512b3747e95a33da3c6e26e2a9f5b4966a" exitCode=0
Nov 03 07:52:46.843478 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:46.842790    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4dgx" event={"ID":"19a8bde5-584f-4ff9-8e96-df33c5cda82d","Type":"ContainerDied","Data":"fd7003fa158b4562f41e538a4a9a79512b3747e95a33da3c6e26e2a9f5b4966a"}
Nov 03 07:52:47.681626 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:47.681546    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:52:47.681806 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:47.681691    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7cmt" podUID="25a7353f-76ad-4ac0-92b8-186377723478"
Nov 03 07:52:47.847722 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:47.847690    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w9c2l" event={"ID":"67ff6855-c974-451d-b582-679e7f26e98d","Type":"ContainerStarted","Data":"8189f22c2732114c58df2b3b6219763949475f1ad6de3bf401fec2a4c4ea24a9"}
Nov 03 07:52:47.850571 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:47.850545    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9j2f9" event={"ID":"676416b5-105e-4650-baed-92baf169a9a6","Type":"ContainerStarted","Data":"a8edfbd48d16f74b5657cff77b9d5dfeccd3507dd066aebbcd8a072d08f96c85"}
Nov 03 07:52:47.864120 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:47.864049    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-w9c2l" podStartSLOduration=3.390957774 podStartE2EDuration="7.864032228s" totalImagesPullingTime="4.473074454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:52:47.863302701 +0000 UTC m=+7.731340422" watchObservedRunningTime="2026-11-03 07:52:47.864032228 +0000 UTC m=+7.732069953"
Nov 03 07:52:47.877948 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:47.877864    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9j2f9" podStartSLOduration=3.896771855 podStartE2EDuration="7.877852218s" totalImagesPullingTime="3.981080363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:52:47.876987958 +0000 UTC m=+7.745025678" watchObservedRunningTime="2026-11-03 07:52:47.877852218 +0000 UTC m=+7.745889940"
Nov 03 07:52:48.682330 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:48.681873    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:52:48.682330 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:48.681992    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6bms" podUID="3a9e7292-b781-4a27-8a48-94f0a8843fc2"
Nov 03 07:52:48.682857 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:48.682689    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:48.682857 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:48.682814    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljbh2" podUID="dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df"
Nov 03 07:52:48.857855 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:48.857583    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db" event={"ID":"5738916f-8202-40ed-823e-90f7c9cc4ec9","Type":"ContainerStarted","Data":"37ae4836852f00cdf0378a242817556fd99fede4156d44b96ef0a27d41757bb2"}
Nov 03 07:52:49.681739 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:49.681717    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:52:49.681886 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:49.681864    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7cmt" podUID="25a7353f-76ad-4ac0-92b8-186377723478"
Nov 03 07:52:49.811524 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:49.811342    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 07:52:49.863376 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:49.863252    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db" event={"ID":"5738916f-8202-40ed-823e-90f7c9cc4ec9","Type":"ContainerStarted","Data":"4ae3294ed80a2b4f350eafec8525bb845bd8c498aecf84fbadcafd14ccc890a3"}
Nov 03 07:52:50.205478 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:50.205444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:50.205655 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:50.205551    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 07:52:50.205655 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:50.205595    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret podName:dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df nodeName:}" failed. No retries permitted until 2026-11-03 07:52:58.205582417 +0000 UTC m=+18.073620129 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret") pod "global-pull-secret-syncer-ljbh2" (UID: "dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df") : object "kube-system"/"original-pull-secret" not registered
Nov 03 07:52:50.307072 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:50.306752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k67j\" (UniqueName: \"kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j\") pod \"network-check-target-c6bms\" (UID: \"3a9e7292-b781-4a27-8a48-94f0a8843fc2\") " pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:52:50.307072 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:50.306849    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs\") pod \"network-metrics-daemon-x7cmt\" (UID: \"25a7353f-76ad-4ac0-92b8-186377723478\") " pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:52:50.307072 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:50.306912    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 07:52:50.307072 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:50.306930    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 07:52:50.307072 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:50.306940    2579 projected.go:196] Error preparing data for projected volume kube-api-access-7k67j for pod openshift-network-diagnostics/network-check-target-c6bms: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:52:50.307072 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:50.306980    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j podName:3a9e7292-b781-4a27-8a48-94f0a8843fc2 nodeName:}" failed. No retries permitted until 2026-11-03 07:52:58.306967895 +0000 UTC m=+18.175005613 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7k67j" (UniqueName: "kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j") pod "network-check-target-c6bms" (UID: "3a9e7292-b781-4a27-8a48-94f0a8843fc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:52:50.307072 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:50.306985    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:52:50.307072 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:50.307038    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs podName:25a7353f-76ad-4ac0-92b8-186377723478 nodeName:}" failed. No retries permitted until 2026-11-03 07:52:58.307021964 +0000 UTC m=+18.175059687 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs") pod "network-metrics-daemon-x7cmt" (UID: "25a7353f-76ad-4ac0-92b8-186377723478") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:52:50.578856 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:50.578638    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T07:52:49.811365542Z","UUID":"ff35ffec-3ef7-4b12-b799-646ac20c727d","Handler":null,"Name":"","Endpoint":""}
Nov 03 07:52:50.584476 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:50.584454    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 07:52:50.584564 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:50.584483    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 07:52:50.683978 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:50.683951    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:50.684061 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:50.683982    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:52:50.684061 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:50.684049    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljbh2" podUID="dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df"
Nov 03 07:52:50.684162 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:50.684121    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6bms" podUID="3a9e7292-b781-4a27-8a48-94f0a8843fc2"
Nov 03 07:52:50.817456 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:50.817411    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9j2f9"
Nov 03 07:52:50.818116 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:50.818098    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9j2f9"
Nov 03 07:52:50.869323 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:50.869257    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-msq9p" event={"ID":"2ed628f2-4323-47b6-ac03-b8567812a740","Type":"ContainerStarted","Data":"62944fedac74e9034ad889fd0259c0754a06780d438640000c678429bf3d6068"}
Nov 03 07:52:51.681928 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:51.681890    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:52:51.682115 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:51.682032    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7cmt" podUID="25a7353f-76ad-4ac0-92b8-186377723478"
Nov 03 07:52:51.875933 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:51.875896    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db" event={"ID":"5738916f-8202-40ed-823e-90f7c9cc4ec9","Type":"ContainerStarted","Data":"1b20df97669339686638095840560cbb88ffc800ab8f02f9baa949621ed46650"}
Nov 03 07:52:51.877852 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:51.877824    2579 generic.go:356] "Generic (PLEG): container finished" podID="19a8bde5-584f-4ff9-8e96-df33c5cda82d" containerID="7c7962c466399ed8a199e725facfbbaeb7216957516a59264a707dae3e5783a9" exitCode=0
Nov 03 07:52:51.877979 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:51.877912    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4dgx" event={"ID":"19a8bde5-584f-4ff9-8e96-df33c5cda82d","Type":"ContainerDied","Data":"7c7962c466399ed8a199e725facfbbaeb7216957516a59264a707dae3e5783a9"}
Nov 03 07:52:51.880534 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:51.880507    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-drdft" event={"ID":"f142ef3a-aa21-48eb-a403-8d51d843c72d","Type":"ContainerStarted","Data":"72fae2c17c4d15292fa59b1431272a5b63709fcef079b769198efa477cf0e167"}
Nov 03 07:52:51.894743 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:51.894683    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db" podStartSLOduration=3.9351196550000003 podStartE2EDuration="11.894673233s" totalImagesPullingTime="7.959553578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:52:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:52:51.894387334 +0000 UTC m=+11.762425059" watchObservedRunningTime="2026-11-03 07:52:51.894673233 +0000 UTC m=+11.762710953"
Nov 03 07:52:51.894913 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:51.894883    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-msq9p" podStartSLOduration=4.472714548 podStartE2EDuration="11.894876396s" totalImagesPullingTime="7.422161848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:52:50.88656434 +0000 UTC m=+10.754602061" watchObservedRunningTime="2026-11-03 07:52:51.894876396 +0000 UTC m=+11.762914118"
Nov 03 07:52:51.909710 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:51.909675    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-drdft" podStartSLOduration=4.067156368 podStartE2EDuration="11.909664116s" totalImagesPullingTime="7.842507748s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:52:51.908819947 +0000 UTC m=+11.776857667" watchObservedRunningTime="2026-11-03 07:52:51.909664116 +0000 UTC m=+11.777701836"
Nov 03 07:52:52.682314 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:52.682092    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:52:52.682436 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:52.682414    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6bms" podUID="3a9e7292-b781-4a27-8a48-94f0a8843fc2"
Nov 03 07:52:52.682717 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:52.682147    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:52.682854 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:52.682821    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljbh2" podUID="dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df"
Nov 03 07:52:52.884841 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:52.884747    2579 generic.go:356] "Generic (PLEG): container finished" podID="19a8bde5-584f-4ff9-8e96-df33c5cda82d" containerID="f544ec45dc8b0ed1730be896018a9530650d5de2a403da0bda714d6c28019c39" exitCode=0
Nov 03 07:52:52.884841 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:52.884795    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4dgx" event={"ID":"19a8bde5-584f-4ff9-8e96-df33c5cda82d","Type":"ContainerDied","Data":"f544ec45dc8b0ed1730be896018a9530650d5de2a403da0bda714d6c28019c39"}
Nov 03 07:52:52.886931 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:52.886910    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kj8jw" event={"ID":"bd5c9498-f849-4dd6-a9e7-27a0d5db4fde","Type":"ContainerStarted","Data":"6e70114e65df09bc14bab40abbda3450c525d5319cab80febdc166f57f7c87b3"}
Nov 03 07:52:52.888440 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:52.888418    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fgwf4" event={"ID":"79888519-e73a-4536-96e4-ce8e3ec99037","Type":"ContainerStarted","Data":"d56783a1659c6b8a3198a2bf5be8a51207ae0dca08ba39711d3c60585cab5982"}
Nov 03 07:52:52.923254 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:52.923213    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kj8jw" podStartSLOduration=5.184235243 podStartE2EDuration="12.923202602s" totalImagesPullingTime="7.738967359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:52:52.922886731 +0000 UTC m=+12.790924449" watchObservedRunningTime="2026-11-03 07:52:52.923202602 +0000 UTC m=+12.791240323"
Nov 03 07:52:53.681857 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:53.681828    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:52:53.681993 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:53.681932    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7cmt" podUID="25a7353f-76ad-4ac0-92b8-186377723478"
Nov 03 07:52:54.681700 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:54.681673    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:52:54.682432 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:54.681792    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6bms" podUID="3a9e7292-b781-4a27-8a48-94f0a8843fc2"
Nov 03 07:52:54.682432 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:54.681849    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:54.682432 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:54.681938    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljbh2" podUID="dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df"
Nov 03 07:52:54.894560 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:54.894530    2579 generic.go:356] "Generic (PLEG): container finished" podID="19a8bde5-584f-4ff9-8e96-df33c5cda82d" containerID="c8a43580ef3f3303a88b4206fa559f3ba85baea124abe36d73b6b07f00678a48" exitCode=0
Nov 03 07:52:54.894741 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:54.894577    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4dgx" event={"ID":"19a8bde5-584f-4ff9-8e96-df33c5cda82d","Type":"ContainerDied","Data":"c8a43580ef3f3303a88b4206fa559f3ba85baea124abe36d73b6b07f00678a48"}
Nov 03 07:52:54.917591 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:54.917548    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fgwf4" podStartSLOduration=5.425503015 podStartE2EDuration="14.91753389s" totalImagesPullingTime="9.492030875s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:52:52.940214446 +0000 UTC m=+12.808252166" watchObservedRunningTime="2026-11-03 07:52:54.91753389 +0000 UTC m=+14.785571611"
Nov 03 07:52:55.682428 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:55.681913    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:52:55.682428 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:55.682060    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7cmt" podUID="25a7353f-76ad-4ac0-92b8-186377723478"
Nov 03 07:52:56.682551 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:56.682508    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:52:56.682979 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:56.682637    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6bms" podUID="3a9e7292-b781-4a27-8a48-94f0a8843fc2"
Nov 03 07:52:56.682979 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:56.682677    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:56.682979 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:56.682748    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljbh2" podUID="dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df"
Nov 03 07:52:57.682482 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:57.682340    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:52:57.682573 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:57.682555    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7cmt" podUID="25a7353f-76ad-4ac0-92b8-186377723478"
Nov 03 07:52:57.904010 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:57.903963    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6d47" event={"ID":"40259634-6b57-492e-8376-75ecf7900b06","Type":"ContainerStarted","Data":"451c1bd4c5f80b1521982e00ff586ced5d5345ce4174d86731f7055fa3ac1aa2"}
Nov 03 07:52:57.904100 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:57.904027    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6d47" event={"ID":"40259634-6b57-492e-8376-75ecf7900b06","Type":"ContainerStarted","Data":"72de608cf3a10bea21ba800f7da735e9c7c725d62a6a1fb2acd004b99a878dfc"}
Nov 03 07:52:57.904100 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:57.904056    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6d47" event={"ID":"40259634-6b57-492e-8376-75ecf7900b06","Type":"ContainerStarted","Data":"d2814de4f8c0ef67707331ed521fbeab954c35855d9618c622142c80659f6bb3"}
Nov 03 07:52:57.904100 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:57.904084    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6d47" event={"ID":"40259634-6b57-492e-8376-75ecf7900b06","Type":"ContainerStarted","Data":"2646e56c619a95d3ff5b1b7233df2ef80fcf36620b66a5a39ba7d797a96e526d"}
Nov 03 07:52:57.904215 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:57.904117    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6d47" event={"ID":"40259634-6b57-492e-8376-75ecf7900b06","Type":"ContainerStarted","Data":"63f52fb526bc745145c85ae4e4365b741518a1e7d8afa67e89240b2950627057"}
Nov 03 07:52:57.906787 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:57.906753    2579 generic.go:356] "Generic (PLEG): container finished" podID="19a8bde5-584f-4ff9-8e96-df33c5cda82d" containerID="0ba8f3f03e7da096ab4560cbc38cc3b90e9e07de7c166bb790acc38c5bf3bc8a" exitCode=0
Nov 03 07:52:57.906860 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:57.906790    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4dgx" event={"ID":"19a8bde5-584f-4ff9-8e96-df33c5cda82d","Type":"ContainerDied","Data":"0ba8f3f03e7da096ab4560cbc38cc3b90e9e07de7c166bb790acc38c5bf3bc8a"}
Nov 03 07:52:58.264515 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:58.264448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:58.264983 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:58.264952    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 07:52:58.265115 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:58.265059    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret podName:dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df nodeName:}" failed. No retries permitted until 2026-11-03 07:53:14.265036432 +0000 UTC m=+34.133074143 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret") pod "global-pull-secret-syncer-ljbh2" (UID: "dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df") : object "kube-system"/"original-pull-secret" not registered
Nov 03 07:52:58.365548 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:58.365522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs\") pod \"network-metrics-daemon-x7cmt\" (UID: \"25a7353f-76ad-4ac0-92b8-186377723478\") " pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:52:58.365680 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:58.365568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k67j\" (UniqueName: \"kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j\") pod \"network-check-target-c6bms\" (UID: \"3a9e7292-b781-4a27-8a48-94f0a8843fc2\") " pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:52:58.365680 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:58.365664    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 07:52:58.365680 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:58.365677    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 07:52:58.365853 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:58.365686    2579 projected.go:196] Error preparing data for projected volume kube-api-access-7k67j for pod openshift-network-diagnostics/network-check-target-c6bms: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:52:58.365853 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:58.365687    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:52:58.365853 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:58.365727    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j podName:3a9e7292-b781-4a27-8a48-94f0a8843fc2 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:14.365715157 +0000 UTC m=+34.233752855 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7k67j" (UniqueName: "kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j") pod "network-check-target-c6bms" (UID: "3a9e7292-b781-4a27-8a48-94f0a8843fc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 07:52:58.365853 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:58.365788    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs podName:25a7353f-76ad-4ac0-92b8-186377723478 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:14.365734286 +0000 UTC m=+34.233772041 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs") pod "network-metrics-daemon-x7cmt" (UID: "25a7353f-76ad-4ac0-92b8-186377723478") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 07:52:58.682337 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:58.682312    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:52:58.682504 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:58.682474    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljbh2" podUID="dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df"
Nov 03 07:52:58.682559 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:58.682518    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:52:58.682621 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:58.682601    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6bms" podUID="3a9e7292-b781-4a27-8a48-94f0a8843fc2"
Nov 03 07:52:58.911486 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:58.911455    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6d47" event={"ID":"40259634-6b57-492e-8376-75ecf7900b06","Type":"ContainerStarted","Data":"cf7b15bd508944c311d18fcf8fb20b0d6c1ee98dfaf684023aa94a6209125147"}
Nov 03 07:52:58.914201 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:58.914180    2579 generic.go:356] "Generic (PLEG): container finished" podID="19a8bde5-584f-4ff9-8e96-df33c5cda82d" containerID="59b9f2e73298e3a7f55a96eedc7ef2eb5c680ab4bcebf6f2d95faab060a8555e" exitCode=0
Nov 03 07:52:58.914318 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:58.914212    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4dgx" event={"ID":"19a8bde5-584f-4ff9-8e96-df33c5cda82d","Type":"ContainerDied","Data":"59b9f2e73298e3a7f55a96eedc7ef2eb5c680ab4bcebf6f2d95faab060a8555e"}
Nov 03 07:52:59.681895 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:59.681865    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:52:59.682058 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:52:59.681975    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7cmt" podUID="25a7353f-76ad-4ac0-92b8-186377723478"
Nov 03 07:52:59.919196 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:59.919167    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-h4dgx" event={"ID":"19a8bde5-584f-4ff9-8e96-df33c5cda82d","Type":"ContainerStarted","Data":"18b60baf52cf6de12c698abfa48c9b7e3e353cb65708d1fa07b3fb8d8d57ed80"}
Nov 03 07:52:59.941842 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:52:59.941742    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-h4dgx" podStartSLOduration=9.077912121 podStartE2EDuration="19.941730931s" totalImagesPullingTime="10.86381881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:52:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:52:59.940056058 +0000 UTC m=+19.808093758" watchObservedRunningTime="2026-11-03 07:52:59.941730931 +0000 UTC m=+19.809768651"
Nov 03 07:53:00.682574 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:00.682541    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:53:00.682726 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:00.682689    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6bms" podUID="3a9e7292-b781-4a27-8a48-94f0a8843fc2"
Nov 03 07:53:00.682803 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:00.682738    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:53:00.682917 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:00.682894    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljbh2" podUID="dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df"
Nov 03 07:53:00.924336 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:00.924307    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6d47" event={"ID":"40259634-6b57-492e-8376-75ecf7900b06","Type":"ContainerStarted","Data":"5cf3f4f70423f502a38dff3b37e3a5c315288e7392441646ea121527bd152e24"}
Nov 03 07:53:01.681397 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:01.681358    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:53:01.681558 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:01.681472    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7cmt" podUID="25a7353f-76ad-4ac0-92b8-186377723478"
Nov 03 07:53:02.681664 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:02.681633    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:53:02.682542 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:02.681669    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:53:02.682542 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:02.681752    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljbh2" podUID="dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df"
Nov 03 07:53:02.682542 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:02.681873    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6bms" podUID="3a9e7292-b781-4a27-8a48-94f0a8843fc2"
Nov 03 07:53:02.930738 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:02.930710    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x6d47" event={"ID":"40259634-6b57-492e-8376-75ecf7900b06","Type":"ContainerStarted","Data":"3de7a1a3e8e807775513dbd903ca27f1e0e6114b2549aee37291b081a2d09baf"}
Nov 03 07:53:02.930980 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:02.930950    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:02.945709 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:02.945663    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:02.956563 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:02.956528    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x6d47" podStartSLOduration=8.200778217 podStartE2EDuration="22.956517507s" totalImagesPullingTime="14.75573929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 07:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:02.956197829 +0000 UTC m=+22.824235550" watchObservedRunningTime="2026-11-03 07:53:02.956517507 +0000 UTC m=+22.824555248"
Nov 03 07:53:03.682532 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:03.682500    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:53:03.682912 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:03.682658    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7cmt" podUID="25a7353f-76ad-4ac0-92b8-186377723478"
Nov 03 07:53:03.933088 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:03.933026    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 07:53:03.933487 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:03.933466    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:03.947716 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:03.947693    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:03.965635 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:03.965612    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c6bms"]
Nov 03 07:53:03.965722 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:03.965711    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:53:03.965818 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:03.965802    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6bms" podUID="3a9e7292-b781-4a27-8a48-94f0a8843fc2"
Nov 03 07:53:03.968854 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:03.968831    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ljbh2"]
Nov 03 07:53:03.968943 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:03.968923    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:53:03.969020 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:03.969003    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljbh2" podUID="dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df"
Nov 03 07:53:03.969470 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:03.969448    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x7cmt"]
Nov 03 07:53:03.969552 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:03.969539    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:53:03.969645 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:03.969629    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7cmt" podUID="25a7353f-76ad-4ac0-92b8-186377723478"
Nov 03 07:53:04.935567 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:04.935537    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 07:53:05.681734 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:05.681697    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:53:05.681922 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:05.681697    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:53:05.681922 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:05.681823    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6bms" podUID="3a9e7292-b781-4a27-8a48-94f0a8843fc2"
Nov 03 07:53:05.681922 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:05.681697    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:53:05.681922 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:05.681881    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljbh2" podUID="dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df"
Nov 03 07:53:05.682058 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:05.681948    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7cmt" podUID="25a7353f-76ad-4ac0-92b8-186377723478"
Nov 03 07:53:05.937352 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:05.937281    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 07:53:07.681956 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:07.681778    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:53:07.682402 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:07.681847    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:53:07.682402 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:07.682025    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6bms" podUID="3a9e7292-b781-4a27-8a48-94f0a8843fc2"
Nov 03 07:53:07.682402 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:07.681854    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:53:07.682402 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:07.682104    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7cmt" podUID="25a7353f-76ad-4ac0-92b8-186377723478"
Nov 03 07:53:07.682402 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:07.682216    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljbh2" podUID="dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df"
Nov 03 07:53:09.681902 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.681863    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:53:09.682286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.681975    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:53:09.682286 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:09.681975    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ljbh2" podUID="dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df"
Nov 03 07:53:09.682286 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.681972    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:53:09.682286 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:09.682053    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c6bms" podUID="3a9e7292-b781-4a27-8a48-94f0a8843fc2"
Nov 03 07:53:09.682286 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:09.682156    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7cmt" podUID="25a7353f-76ad-4ac0-92b8-186377723478"
Nov 03 07:53:09.875215 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.875189    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeReady"
Nov 03 07:53:09.875361 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.875299    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 07:53:09.918101 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.918024    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kr2q8"]
Nov 03 07:53:09.921319 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.921297    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nwtkn"]
Nov 03 07:53:09.921577 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.921548    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kr2q8"
Nov 03 07:53:09.924305 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.924286    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-v2z4k"]
Nov 03 07:53:09.924405 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.924376    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nwtkn"
Nov 03 07:53:09.926949 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.926932    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-grgjj"]
Nov 03 07:53:09.927037 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.926982    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-v2z4k"
Nov 03 07:53:09.927859 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.927828    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 07:53:09.928123 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.928100    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 07:53:09.930472 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.930456    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74774885c5-97jxz"]
Nov 03 07:53:09.930594 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.930576    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-grgjj"
Nov 03 07:53:09.933253 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.933238    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67ffc88546-z24dr"]
Nov 03 07:53:09.933359 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.933343    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:09.937781 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.937436    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"]
Nov 03 07:53:09.937867 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.937852    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:09.945994 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.944321    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-crn8p"]
Nov 03 07:53:09.946551 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.946519    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:09.947316 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.947296    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 07:53:09.947401 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.947324    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 07:53:09.947401 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.947397    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 07:53:09.947873 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.947805    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-28mc2\""
Nov 03 07:53:09.947987 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.947899    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 07:53:09.947987 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.947946    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-cg8rh\""
Nov 03 07:53:09.949638 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.948724    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"]
Nov 03 07:53:09.949638 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.949006    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:09.952555 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.952539    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-79698c9d55-m2lbh"]
Nov 03 07:53:09.952655 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.952641    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:09.955830 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.955812    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk"]
Nov 03 07:53:09.956161 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.956146    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:09.960682 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.960661    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"]
Nov 03 07:53:09.960790 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.960730    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk"
Nov 03 07:53:09.966177 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.966160    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f"]
Nov 03 07:53:09.966957 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.966935    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:09.969148 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.969130    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fshz5"]
Nov 03 07:53:09.969301 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.969284    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f"
Nov 03 07:53:09.972018 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.972002    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zhh6g"]
Nov 03 07:53:09.972120 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.972106    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fshz5"
Nov 03 07:53:09.974875 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.974859    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-v2z4k"]
Nov 03 07:53:09.974979 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.974885    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nwtkn"]
Nov 03 07:53:09.974979 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.974900    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-grgjj"]
Nov 03 07:53:09.974979 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.974915    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kr2q8"]
Nov 03 07:53:09.974979 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.974934    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74774885c5-97jxz"]
Nov 03 07:53:09.974979 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.974953    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"]
Nov 03 07:53:09.974979 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.974972    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67ffc88546-z24dr"]
Nov 03 07:53:09.974979 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.974973    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zhh6g"
Nov 03 07:53:09.975318 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.974992    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79698c9d55-m2lbh"]
Nov 03 07:53:09.975318 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.975011    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f"]
Nov 03 07:53:09.975318 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.975032    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fshz5"]
Nov 03 07:53:09.975318 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.975058    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-crn8p"]
Nov 03 07:53:09.975318 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.975079    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"]
Nov 03 07:53:09.975318 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.975103    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"]
Nov 03 07:53:09.975318 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.975125    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk"]
Nov 03 07:53:09.975318 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:09.975160    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zhh6g"]
Nov 03 07:53:10.028516 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.028496    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 07:53:10.028724 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.028705    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 07:53:10.028872 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.028727    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 07:53:10.028872 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.028864    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 07:53:10.028872 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.028735    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 07:53:10.029035 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.028839    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 07:53:10.029035 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.028948    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 07:53:10.029035 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.028949    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 07:53:10.029035 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.028952    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 07:53:10.029035 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.029021    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 07:53:10.029207 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.029088    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 07:53:10.029207 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.029113    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 07:53:10.029207 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.029116    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 07:53:10.029207 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.029156    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 07:53:10.029207 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.029158    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 07:53:10.029207 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.029187    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 07:53:10.029207 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.029200    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 07:53:10.030277 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.030255    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 07:53:10.030277 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.030271    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 07:53:10.030412 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.030280    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 07:53:10.030412 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.030263    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 07:53:10.030412 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.030257    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 07:53:10.030412 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.030262    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 07:53:10.030412 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.030283    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 07:53:10.030412 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.030327    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 07:53:10.039135 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.039120    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 07:53:10.040263 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.040248    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6ldkm"]
Nov 03 07:53:10.043072 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.043053    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6ldkm"
Nov 03 07:53:10.046932 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.046914    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 07:53:10.047026 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.046951    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 07:53:10.047889 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.047746    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-46l5d"]
Nov 03 07:53:10.048596 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.048574    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 07:53:10.048677 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.048644    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 07:53:10.048724 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.048696    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 07:53:10.048830 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.048812    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 07:53:10.048830 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.048827    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 07:53:10.048989 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.048815    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 07:53:10.048989 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.048853    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 07:53:10.048989 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.048873    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 07:53:10.048989 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.048880    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 07:53:10.049689 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.048995    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 07:53:10.050793 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.050680    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:10.051562 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.051543    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6ldkm"]
Nov 03 07:53:10.052044 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.052020    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-46l5d"]
Nov 03 07:53:10.055083 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055062    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44str\" (UniqueName: \"kubernetes.io/projected/dc162541-2ab5-4475-8856-850a7c5fd437-kube-api-access-44str\") pod \"cluster-samples-operator-6ddb6c8b94-dtgx2\" (UID: \"dc162541-2ab5-4475-8856-850a7c5fd437\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:10.055177 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055093    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c620e360-6548-4b0c-ba9a-a9150cf9939c-service-ca-bundle\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:10.055177 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055108    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd8qs\" (UniqueName: \"kubernetes.io/projected/c620e360-6548-4b0c-ba9a-a9150cf9939c-kube-api-access-sd8qs\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:10.055177 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg5vz\" (UniqueName: \"kubernetes.io/projected/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3-kube-api-access-kg5vz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk\" (UID: \"e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk"
Nov 03 07:53:10.055177 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk\" (UID: \"e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk"
Nov 03 07:53:10.055314 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055200    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2l9g\" (UniqueName: \"kubernetes.io/projected/20b9100c-2539-4227-8f2a-c93660540664-kube-api-access-v2l9g\") pod \"kas-connection-checker-685895c7c9-v2z4k\" (UID: \"20b9100c-2539-4227-8f2a-c93660540664\") " pod="kube-system/kas-connection-checker-685895c7c9-v2z4k"
Nov 03 07:53:10.055314 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055250    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/119141fd-3c4f-4b94-9f18-6892986867e9-image-registry-private-configuration\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.055314 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/751205b0-8312-4ea6-920c-9517e19566f2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-grgjj\" (UID: \"751205b0-8312-4ea6-920c-9517e19566f2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grgjj"
Nov 03 07:53:10.055403 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055335    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/119141fd-3c4f-4b94-9f18-6892986867e9-installation-pull-secrets\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.055403 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkn8t\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-kube-api-access-vkn8t\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.055464 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055430    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24tfh\" (UniqueName: \"kubernetes.io/projected/08817fd9-0879-4b77-a767-5f4c4ecf8d5b-kube-api-access-24tfh\") pod \"kas-connection-checker-685895c7c9-nwtkn\" (UID: \"08817fd9-0879-4b77-a767-5f4c4ecf8d5b\") " pod="kube-system/kas-connection-checker-685895c7c9-nwtkn"
Nov 03 07:53:10.055464 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzxps\" (UniqueName: \"kubernetes.io/projected/3cb721ff-442b-41b7-8287-220dbfbb41da-kube-api-access-pzxps\") pod \"volume-data-source-validator-58c9f5f46d-zhh6g\" (UID: \"3cb721ff-442b-41b7-8287-220dbfbb41da\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zhh6g"
Nov 03 07:53:10.055579 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055564    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvhx7\" (UniqueName: \"kubernetes.io/projected/edf55612-a9f0-4fd5-ab00-ac2271acfb30-kube-api-access-gvhx7\") pod \"kas-connection-checker-685895c7c9-kr2q8\" (UID: \"edf55612-a9f0-4fd5-ab00-ac2271acfb30\") " pod="kube-system/kas-connection-checker-685895c7c9-kr2q8"
Nov 03 07:53:10.055613 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055599    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhpl8\" (UniqueName: \"kubernetes.io/projected/7e349708-a596-40ca-8724-6c2169cace8c-kube-api-access-rhpl8\") pod \"network-check-source-6f78d59f99-fshz5\" (UID: \"7e349708-a596-40ca-8724-6c2169cace8c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fshz5"
Nov 03 07:53:10.055645 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/58448f55-1670-41ce-8e35-182fe94f2d7e-config\") pod \"service-ca-operator-544d749c8c-gt59f\" (UID: \"58448f55-1670-41ce-8e35-182fe94f2d7e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f"
Nov 03 07:53:10.055697 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055683    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df604a27-2a06-42a0-bcbd-f7a10c1420c1-trusted-ca\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.055738 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055724    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wwkr\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-kube-api-access-4wwkr\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.055785 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055752    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppt2q\" (UniqueName: \"kubernetes.io/projected/d83648de-81dd-4160-8fa9-06d36b38d9de-kube-api-access-ppt2q\") pod \"console-operator-7cd56b8dbc-5qpnb\" (UID: \"d83648de-81dd-4160-8fa9-06d36b38d9de\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:10.055821 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-certificates\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.055860 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dtgx2\" (UID: \"dc162541-2ab5-4475-8856-850a7c5fd437\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:10.055893 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df604a27-2a06-42a0-bcbd-f7a10c1420c1-ca-trust-extracted\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.056012 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.055989    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-f68ks\" (UID: \"c383d3eb-10c7-4fe5-99b6-04c64472ef0a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:10.056118 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.056101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/119141fd-3c4f-4b94-9f18-6892986867e9-ca-trust-extracted\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.056171 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.056144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-stats-auth\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:10.056218 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.056188    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz98m\" (UniqueName: \"kubernetes.io/projected/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-kube-api-access-nz98m\") pod \"cluster-monitoring-operator-884576c57-f68ks\" (UID: \"c383d3eb-10c7-4fe5-99b6-04c64472ef0a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:10.056321 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.056291    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvqfg\" (UniqueName: \"kubernetes.io/projected/ce890601-127d-4995-9487-635e7aab04f8-kube-api-access-jvqfg\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:10.056395 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.056345    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk\" (UID: \"e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk"
Nov 03 07:53:10.056444 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.056424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.056489 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.056467    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:10.056540 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.056488    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d83648de-81dd-4160-8fa9-06d36b38d9de-trusted-ca\") pod \"console-operator-7cd56b8dbc-5qpnb\" (UID: \"d83648de-81dd-4160-8fa9-06d36b38d9de\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:10.056592 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.056541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-f68ks\" (UID: \"c383d3eb-10c7-4fe5-99b6-04c64472ef0a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:10.056592 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.056579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv6bs\" (UniqueName: \"kubernetes.io/projected/58448f55-1670-41ce-8e35-182fe94f2d7e-kube-api-access-bv6bs\") pod \"service-ca-operator-544d749c8c-gt59f\" (UID: \"58448f55-1670-41ce-8e35-182fe94f2d7e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f"
Nov 03 07:53:10.056685 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.056650    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/119141fd-3c4f-4b94-9f18-6892986867e9-trusted-ca\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.056739 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.056681    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dtgx2\" (UID: \"dc162541-2ab5-4475-8856-850a7c5fd437\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:10.056814 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.056742    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df604a27-2a06-42a0-bcbd-f7a10c1420c1-image-registry-private-configuration\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.056814 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.056803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.056889 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.056846    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c620e360-6548-4b0c-ba9a-a9150cf9939c-tmp\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:10.056889 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.056868    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df604a27-2a06-42a0-bcbd-f7a10c1420c1-installation-pull-secrets\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.056889 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.056884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d83648de-81dd-4160-8fa9-06d36b38d9de-serving-cert\") pod \"console-operator-7cd56b8dbc-5qpnb\" (UID: \"d83648de-81dd-4160-8fa9-06d36b38d9de\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:10.056989 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.056942    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-bound-sa-token\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.056989 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.056962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58448f55-1670-41ce-8e35-182fe94f2d7e-serving-cert\") pod \"service-ca-operator-544d749c8c-gt59f\" (UID: \"58448f55-1670-41ce-8e35-182fe94f2d7e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f"
Nov 03 07:53:10.057041 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.057014    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c620e360-6548-4b0c-ba9a-a9150cf9939c-snapshots\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:10.057071 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.057053    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/751205b0-8312-4ea6-920c-9517e19566f2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-grgjj\" (UID: \"751205b0-8312-4ea6-920c-9517e19566f2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grgjj"
Nov 03 07:53:10.057152 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.057089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-metrics-certs\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:10.057152 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.057113    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/119141fd-3c4f-4b94-9f18-6892986867e9-registry-certificates\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.057254 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.057161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-default-certificate\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:10.057254 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.057199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c620e360-6548-4b0c-ba9a-a9150cf9939c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:10.057254 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.057228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-bound-sa-token\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.057354 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.057264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c620e360-6548-4b0c-ba9a-a9150cf9939c-serving-cert\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:10.057354 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.057298    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d83648de-81dd-4160-8fa9-06d36b38d9de-config\") pod \"console-operator-7cd56b8dbc-5qpnb\" (UID: \"d83648de-81dd-4160-8fa9-06d36b38d9de\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:10.127403 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.127375    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 07:53:10.127575 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.127520    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 07:53:10.127575 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.127546    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 07:53:10.142912 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.142890    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 07:53:10.143072 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.143050    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 07:53:10.143151 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.143140    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjbhz\""
Nov 03 07:53:10.143193 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.143168    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ghck6\""
Nov 03 07:53:10.157846 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.157825    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:10.157954 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.157858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d83648de-81dd-4160-8fa9-06d36b38d9de-trusted-ca\") pod \"console-operator-7cd56b8dbc-5qpnb\" (UID: \"d83648de-81dd-4160-8fa9-06d36b38d9de\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:10.157954 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.157886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-f68ks\" (UID: \"c383d3eb-10c7-4fe5-99b6-04c64472ef0a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:10.157954 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.157911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv6bs\" (UniqueName: \"kubernetes.io/projected/58448f55-1670-41ce-8e35-182fe94f2d7e-kube-api-access-bv6bs\") pod \"service-ca-operator-544d749c8c-gt59f\" (UID: \"58448f55-1670-41ce-8e35-182fe94f2d7e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f"
Nov 03 07:53:10.157954 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.157937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/119141fd-3c4f-4b94-9f18-6892986867e9-trusted-ca\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.158152 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.157959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dtgx2\" (UID: \"dc162541-2ab5-4475-8856-850a7c5fd437\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:10.158152 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.157986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df604a27-2a06-42a0-bcbd-f7a10c1420c1-image-registry-private-configuration\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.158152 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.158001    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 07:53:10.158152 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.158001    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle podName:ce890601-127d-4995-9487-635e7aab04f8 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:10.657980678 +0000 UTC m=+30.526018405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle") pod "router-default-79698c9d55-m2lbh" (UID: "ce890601-127d-4995-9487-635e7aab04f8") : configmap references non-existent config key: service-ca.crt
Nov 03 07:53:10.158152 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.158090    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-cluster-monitoring-operator-tls podName:c383d3eb-10c7-4fe5-99b6-04c64472ef0a nodeName:}" failed. No retries permitted until 2026-11-03 07:53:10.658071106 +0000 UTC m=+30.526108822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-f68ks" (UID: "c383d3eb-10c7-4fe5-99b6-04c64472ef0a") : secret "cluster-monitoring-operator-tls" not found
Nov 03 07:53:10.158152 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.158152 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp9jm\" (UniqueName: \"kubernetes.io/projected/279b3dd8-58be-43a1-9994-59fa434409f5-kube-api-access-lp9jm\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:10.158456 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c620e360-6548-4b0c-ba9a-a9150cf9939c-tmp\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:10.158456 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df604a27-2a06-42a0-bcbd-f7a10c1420c1-installation-pull-secrets\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.158456 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158248    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d83648de-81dd-4160-8fa9-06d36b38d9de-serving-cert\") pod \"console-operator-7cd56b8dbc-5qpnb\" (UID: \"d83648de-81dd-4160-8fa9-06d36b38d9de\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:10.158456 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-bound-sa-token\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.158456 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.158296    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 07:53:10.158456 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.158310    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74774885c5-97jxz: secret "image-registry-tls" not found
Nov 03 07:53:10.158456 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58448f55-1670-41ce-8e35-182fe94f2d7e-serving-cert\") pod \"service-ca-operator-544d749c8c-gt59f\" (UID: \"58448f55-1670-41ce-8e35-182fe94f2d7e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f"
Nov 03 07:53:10.158456 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c620e360-6548-4b0c-ba9a-a9150cf9939c-snapshots\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:10.158456 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/279b3dd8-58be-43a1-9994-59fa434409f5-metrics-tls\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:10.158456 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158396    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/751205b0-8312-4ea6-920c-9517e19566f2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-grgjj\" (UID: \"751205b0-8312-4ea6-920c-9517e19566f2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grgjj"
Nov 03 07:53:10.158456 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158437    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-metrics-certs\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:10.158941 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/279b3dd8-58be-43a1-9994-59fa434409f5-config-volume\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:10.158941 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/119141fd-3c4f-4b94-9f18-6892986867e9-registry-certificates\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.158941 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-default-certificate\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:10.158941 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c620e360-6548-4b0c-ba9a-a9150cf9939c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:10.158941 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-bound-sa-token\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.158941 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c620e360-6548-4b0c-ba9a-a9150cf9939c-serving-cert\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:10.158941 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d83648de-81dd-4160-8fa9-06d36b38d9de-config\") pod \"console-operator-7cd56b8dbc-5qpnb\" (UID: \"d83648de-81dd-4160-8fa9-06d36b38d9de\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:10.158941 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158629    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c620e360-6548-4b0c-ba9a-a9150cf9939c-tmp\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:10.158941 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44str\" (UniqueName: \"kubernetes.io/projected/dc162541-2ab5-4475-8856-850a7c5fd437-kube-api-access-44str\") pod \"cluster-samples-operator-6ddb6c8b94-dtgx2\" (UID: \"dc162541-2ab5-4475-8856-850a7c5fd437\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:10.158941 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c620e360-6548-4b0c-ba9a-a9150cf9939c-service-ca-bundle\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:10.158941 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158721    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd8qs\" (UniqueName: \"kubernetes.io/projected/c620e360-6548-4b0c-ba9a-a9150cf9939c-kube-api-access-sd8qs\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:10.158941 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg5vz\" (UniqueName: \"kubernetes.io/projected/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3-kube-api-access-kg5vz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk\" (UID: \"e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk"
Nov 03 07:53:10.158941 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk\" (UID: \"e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk"
Nov 03 07:53:10.158941 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279b3dd8-58be-43a1-9994-59fa434409f5-tmp-dir\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:10.158941 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158885    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2l9g\" (UniqueName: \"kubernetes.io/projected/20b9100c-2539-4227-8f2a-c93660540664-kube-api-access-v2l9g\") pod \"kas-connection-checker-685895c7c9-v2z4k\" (UID: \"20b9100c-2539-4227-8f2a-c93660540664\") " pod="kube-system/kas-connection-checker-685895c7c9-v2z4k"
Nov 03 07:53:10.158941 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158926    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/119141fd-3c4f-4b94-9f18-6892986867e9-image-registry-private-configuration\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.159572 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/751205b0-8312-4ea6-920c-9517e19566f2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-grgjj\" (UID: \"751205b0-8312-4ea6-920c-9517e19566f2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grgjj"
Nov 03 07:53:10.159572 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.158986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/119141fd-3c4f-4b94-9f18-6892986867e9-installation-pull-secrets\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.159572 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.159013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkn8t\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-kube-api-access-vkn8t\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.159572 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.159074    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24tfh\" (UniqueName: \"kubernetes.io/projected/08817fd9-0879-4b77-a767-5f4c4ecf8d5b-kube-api-access-24tfh\") pod \"kas-connection-checker-685895c7c9-nwtkn\" (UID: \"08817fd9-0879-4b77-a767-5f4c4ecf8d5b\") " pod="kube-system/kas-connection-checker-685895c7c9-nwtkn"
Nov 03 07:53:10.159572 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.159105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzxps\" (UniqueName: \"kubernetes.io/projected/3cb721ff-442b-41b7-8287-220dbfbb41da-kube-api-access-pzxps\") pod \"volume-data-source-validator-58c9f5f46d-zhh6g\" (UID: \"3cb721ff-442b-41b7-8287-220dbfbb41da\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zhh6g"
Nov 03 07:53:10.159572 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.159143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvhx7\" (UniqueName: \"kubernetes.io/projected/edf55612-a9f0-4fd5-ab00-ac2271acfb30-kube-api-access-gvhx7\") pod \"kas-connection-checker-685895c7c9-kr2q8\" (UID: \"edf55612-a9f0-4fd5-ab00-ac2271acfb30\") " pod="kube-system/kas-connection-checker-685895c7c9-kr2q8"
Nov 03 07:53:10.159572 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.159170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhpl8\" (UniqueName: \"kubernetes.io/projected/7e349708-a596-40ca-8724-6c2169cace8c-kube-api-access-rhpl8\") pod \"network-check-source-6f78d59f99-fshz5\" (UID: \"7e349708-a596-40ca-8724-6c2169cace8c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fshz5"
Nov 03 07:53:10.159572 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.159200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/58448f55-1670-41ce-8e35-182fe94f2d7e-config\") pod \"service-ca-operator-544d749c8c-gt59f\" (UID: \"58448f55-1670-41ce-8e35-182fe94f2d7e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f"
Nov 03 07:53:10.159572 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.159228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df604a27-2a06-42a0-bcbd-f7a10c1420c1-trusted-ca\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.159572 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.159289    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/119141fd-3c4f-4b94-9f18-6892986867e9-trusted-ca\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.159572 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.159430    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dtgx2\" (UID: \"dc162541-2ab5-4475-8856-850a7c5fd437\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:10.159572 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.159542    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d83648de-81dd-4160-8fa9-06d36b38d9de-trusted-ca\") pod \"console-operator-7cd56b8dbc-5qpnb\" (UID: \"d83648de-81dd-4160-8fa9-06d36b38d9de\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:10.160202 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.159670    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c620e360-6548-4b0c-ba9a-a9150cf9939c-service-ca-bundle\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:10.160202 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.160123    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df604a27-2a06-42a0-bcbd-f7a10c1420c1-trusted-ca\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.161415 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.158957    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 07:53:10.161415 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.160398    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c620e360-6548-4b0c-ba9a-a9150cf9939c-snapshots\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:10.161415 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.160452    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk\" (UID: \"e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk"
Nov 03 07:53:10.161415 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.159009    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 07:53:10.161415 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.159958    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls podName:df604a27-2a06-42a0-bcbd-f7a10c1420c1 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:10.659938314 +0000 UTC m=+30.527976013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls") pod "image-registry-74774885c5-97jxz" (UID: "df604a27-2a06-42a0-bcbd-f7a10c1420c1") : secret "image-registry-tls" not found
Nov 03 07:53:10.161415 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.160557    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/751205b0-8312-4ea6-920c-9517e19566f2-networking-console-plugin-cert podName:751205b0-8312-4ea6-920c-9517e19566f2 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:10.660541012 +0000 UTC m=+30.528578716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/751205b0-8312-4ea6-920c-9517e19566f2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-grgjj" (UID: "751205b0-8312-4ea6-920c-9517e19566f2") : secret "networking-console-plugin-cert" not found
Nov 03 07:53:10.161415 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.160570    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/119141fd-3c4f-4b94-9f18-6892986867e9-registry-certificates\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.161415 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.160580    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wwkr\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-kube-api-access-4wwkr\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.161415 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.160640    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-metrics-certs podName:ce890601-127d-4995-9487-635e7aab04f8 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:10.660628443 +0000 UTC m=+30.528666150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-metrics-certs") pod "router-default-79698c9d55-m2lbh" (UID: "ce890601-127d-4995-9487-635e7aab04f8") : secret "router-metrics-certs-default" not found
Nov 03 07:53:10.161415 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.161069    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d83648de-81dd-4160-8fa9-06d36b38d9de-config\") pod \"console-operator-7cd56b8dbc-5qpnb\" (UID: \"d83648de-81dd-4160-8fa9-06d36b38d9de\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:10.161415 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.161069    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/751205b0-8312-4ea6-920c-9517e19566f2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-grgjj\" (UID: \"751205b0-8312-4ea6-920c-9517e19566f2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grgjj"
Nov 03 07:53:10.161415 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.161212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppt2q\" (UniqueName: \"kubernetes.io/projected/d83648de-81dd-4160-8fa9-06d36b38d9de-kube-api-access-ppt2q\") pod \"console-operator-7cd56b8dbc-5qpnb\" (UID: \"d83648de-81dd-4160-8fa9-06d36b38d9de\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:10.161415 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.161249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-certificates\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.161415 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.161279    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvnh8\" (UniqueName: \"kubernetes.io/projected/78b91dbb-d9e9-4524-8827-d74c8c0625d0-kube-api-access-kvnh8\") pod \"ingress-canary-6ldkm\" (UID: \"78b91dbb-d9e9-4524-8827-d74c8c0625d0\") " pod="openshift-ingress-canary/ingress-canary-6ldkm"
Nov 03 07:53:10.161415 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.161311    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dtgx2\" (UID: \"dc162541-2ab5-4475-8856-850a7c5fd437\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:10.162276 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.161339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df604a27-2a06-42a0-bcbd-f7a10c1420c1-ca-trust-extracted\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.162276 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.161371    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-f68ks\" (UID: \"c383d3eb-10c7-4fe5-99b6-04c64472ef0a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:10.162276 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.161400    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78b91dbb-d9e9-4524-8827-d74c8c0625d0-cert\") pod \"ingress-canary-6ldkm\" (UID: \"78b91dbb-d9e9-4524-8827-d74c8c0625d0\") " pod="openshift-ingress-canary/ingress-canary-6ldkm"
Nov 03 07:53:10.162276 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.161429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/119141fd-3c4f-4b94-9f18-6892986867e9-ca-trust-extracted\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.162276 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.161458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-stats-auth\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:10.162276 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.161484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz98m\" (UniqueName: \"kubernetes.io/projected/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-kube-api-access-nz98m\") pod \"cluster-monitoring-operator-884576c57-f68ks\" (UID: \"c383d3eb-10c7-4fe5-99b6-04c64472ef0a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:10.162276 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.161514    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvqfg\" (UniqueName: \"kubernetes.io/projected/ce890601-127d-4995-9487-635e7aab04f8-kube-api-access-jvqfg\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:10.162276 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.161541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk\" (UID: \"e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk"
Nov 03 07:53:10.162276 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.161575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.162276 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.161573    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/58448f55-1670-41ce-8e35-182fe94f2d7e-config\") pod \"service-ca-operator-544d749c8c-gt59f\" (UID: \"58448f55-1670-41ce-8e35-182fe94f2d7e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f"
Nov 03 07:53:10.162276 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.161692    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c620e360-6548-4b0c-ba9a-a9150cf9939c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:10.162276 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.161990    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/119141fd-3c4f-4b94-9f18-6892986867e9-ca-trust-extracted\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.162276 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.161997    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-certificates\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.162276 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.162094    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 07:53:10.162276 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.162134    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-tls podName:dc162541-2ab5-4475-8856-850a7c5fd437 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:10.662121324 +0000 UTC m=+30.530159039 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dtgx2" (UID: "dc162541-2ab5-4475-8856-850a7c5fd437") : secret "samples-operator-tls" not found
Nov 03 07:53:10.163127 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.162402    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df604a27-2a06-42a0-bcbd-f7a10c1420c1-ca-trust-extracted\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.163127 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.162990    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 07:53:10.163127 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.163007    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67ffc88546-z24dr: secret "image-registry-tls" not found
Nov 03 07:53:10.163127 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.163056    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls podName:119141fd-3c4f-4b94-9f18-6892986867e9 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:10.663041861 +0000 UTC m=+30.531079576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls") pod "image-registry-67ffc88546-z24dr" (UID: "119141fd-3c4f-4b94-9f18-6892986867e9") : secret "image-registry-tls" not found
Nov 03 07:53:10.163127 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.163105    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-f68ks\" (UID: \"c383d3eb-10c7-4fe5-99b6-04c64472ef0a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:10.163426 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.163328    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58448f55-1670-41ce-8e35-182fe94f2d7e-serving-cert\") pod \"service-ca-operator-544d749c8c-gt59f\" (UID: \"58448f55-1670-41ce-8e35-182fe94f2d7e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f"
Nov 03 07:53:10.163506 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.163454    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d83648de-81dd-4160-8fa9-06d36b38d9de-serving-cert\") pod \"console-operator-7cd56b8dbc-5qpnb\" (UID: \"d83648de-81dd-4160-8fa9-06d36b38d9de\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:10.163850 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.163815    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/119141fd-3c4f-4b94-9f18-6892986867e9-image-registry-private-configuration\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.164119 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.164098    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df604a27-2a06-42a0-bcbd-f7a10c1420c1-installation-pull-secrets\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.164534 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.164489    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-stats-auth\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:10.165389 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.165172    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/119141fd-3c4f-4b94-9f18-6892986867e9-installation-pull-secrets\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.165466 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.165443    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk\" (UID: \"e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk"
Nov 03 07:53:10.165598 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.165574    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df604a27-2a06-42a0-bcbd-f7a10c1420c1-image-registry-private-configuration\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.165897 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.165874    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c620e360-6548-4b0c-ba9a-a9150cf9939c-serving-cert\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:10.166277 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.166253    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-default-certificate\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:10.166670 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.166647    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv6bs\" (UniqueName: \"kubernetes.io/projected/58448f55-1670-41ce-8e35-182fe94f2d7e-kube-api-access-bv6bs\") pod \"service-ca-operator-544d749c8c-gt59f\" (UID: \"58448f55-1670-41ce-8e35-182fe94f2d7e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f"
Nov 03 07:53:10.173679 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.173625    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhpl8\" (UniqueName: \"kubernetes.io/projected/7e349708-a596-40ca-8724-6c2169cace8c-kube-api-access-rhpl8\") pod \"network-check-source-6f78d59f99-fshz5\" (UID: \"7e349708-a596-40ca-8724-6c2169cace8c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fshz5"
Nov 03 07:53:10.174250 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.174225    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd8qs\" (UniqueName: \"kubernetes.io/projected/c620e360-6548-4b0c-ba9a-a9150cf9939c-kube-api-access-sd8qs\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:10.174337 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.174274    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvqfg\" (UniqueName: \"kubernetes.io/projected/ce890601-127d-4995-9487-635e7aab04f8-kube-api-access-jvqfg\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:10.174648 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.174624    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-bound-sa-token\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.174834 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.174814    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkn8t\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-kube-api-access-vkn8t\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.174927 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.174875    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg5vz\" (UniqueName: \"kubernetes.io/projected/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3-kube-api-access-kg5vz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk\" (UID: \"e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk"
Nov 03 07:53:10.175063 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.175041    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvhx7\" (UniqueName: \"kubernetes.io/projected/edf55612-a9f0-4fd5-ab00-ac2271acfb30-kube-api-access-gvhx7\") pod \"kas-connection-checker-685895c7c9-kr2q8\" (UID: \"edf55612-a9f0-4fd5-ab00-ac2271acfb30\") " pod="kube-system/kas-connection-checker-685895c7c9-kr2q8"
Nov 03 07:53:10.175420 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.175400    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wwkr\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-kube-api-access-4wwkr\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.175487 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.175402    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-bound-sa-token\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.175487 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.175482    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44str\" (UniqueName: \"kubernetes.io/projected/dc162541-2ab5-4475-8856-850a7c5fd437-kube-api-access-44str\") pod \"cluster-samples-operator-6ddb6c8b94-dtgx2\" (UID: \"dc162541-2ab5-4475-8856-850a7c5fd437\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:10.175797 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.175770    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24tfh\" (UniqueName: \"kubernetes.io/projected/08817fd9-0879-4b77-a767-5f4c4ecf8d5b-kube-api-access-24tfh\") pod \"kas-connection-checker-685895c7c9-nwtkn\" (UID: \"08817fd9-0879-4b77-a767-5f4c4ecf8d5b\") " pod="kube-system/kas-connection-checker-685895c7c9-nwtkn"
Nov 03 07:53:10.177107 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.177087    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2l9g\" (UniqueName: \"kubernetes.io/projected/20b9100c-2539-4227-8f2a-c93660540664-kube-api-access-v2l9g\") pod \"kas-connection-checker-685895c7c9-v2z4k\" (UID: \"20b9100c-2539-4227-8f2a-c93660540664\") " pod="kube-system/kas-connection-checker-685895c7c9-v2z4k"
Nov 03 07:53:10.177207 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.177128    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz98m\" (UniqueName: \"kubernetes.io/projected/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-kube-api-access-nz98m\") pod \"cluster-monitoring-operator-884576c57-f68ks\" (UID: \"c383d3eb-10c7-4fe5-99b6-04c64472ef0a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:10.177207 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.177198    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppt2q\" (UniqueName: \"kubernetes.io/projected/d83648de-81dd-4160-8fa9-06d36b38d9de-kube-api-access-ppt2q\") pod \"console-operator-7cd56b8dbc-5qpnb\" (UID: \"d83648de-81dd-4160-8fa9-06d36b38d9de\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:10.177354 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.177336    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzxps\" (UniqueName: \"kubernetes.io/projected/3cb721ff-442b-41b7-8287-220dbfbb41da-kube-api-access-pzxps\") pod \"volume-data-source-validator-58c9f5f46d-zhh6g\" (UID: \"3cb721ff-442b-41b7-8287-220dbfbb41da\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zhh6g"
Nov 03 07:53:10.237456 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.237438    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kr2q8"
Nov 03 07:53:10.250044 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.250001    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-nwtkn"
Nov 03 07:53:10.259775 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.259745    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-v2z4k"
Nov 03 07:53:10.262036 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.262011    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp9jm\" (UniqueName: \"kubernetes.io/projected/279b3dd8-58be-43a1-9994-59fa434409f5-kube-api-access-lp9jm\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:10.262114 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.262059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/279b3dd8-58be-43a1-9994-59fa434409f5-metrics-tls\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:10.262166 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.262121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/279b3dd8-58be-43a1-9994-59fa434409f5-config-volume\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:10.262218 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.262166    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279b3dd8-58be-43a1-9994-59fa434409f5-tmp-dir\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:10.262264 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.262226    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 07:53:10.262264 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.262242    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvnh8\" (UniqueName: \"kubernetes.io/projected/78b91dbb-d9e9-4524-8827-d74c8c0625d0-kube-api-access-kvnh8\") pod \"ingress-canary-6ldkm\" (UID: \"78b91dbb-d9e9-4524-8827-d74c8c0625d0\") " pod="openshift-ingress-canary/ingress-canary-6ldkm"
Nov 03 07:53:10.262357 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.262284    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/279b3dd8-58be-43a1-9994-59fa434409f5-metrics-tls podName:279b3dd8-58be-43a1-9994-59fa434409f5 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:10.762266318 +0000 UTC m=+30.630304022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/279b3dd8-58be-43a1-9994-59fa434409f5-metrics-tls") pod "dns-default-46l5d" (UID: "279b3dd8-58be-43a1-9994-59fa434409f5") : secret "dns-default-metrics-tls" not found
Nov 03 07:53:10.262428 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.262410    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78b91dbb-d9e9-4524-8827-d74c8c0625d0-cert\") pod \"ingress-canary-6ldkm\" (UID: \"78b91dbb-d9e9-4524-8827-d74c8c0625d0\") " pod="openshift-ingress-canary/ingress-canary-6ldkm"
Nov 03 07:53:10.262568 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.262551    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279b3dd8-58be-43a1-9994-59fa434409f5-tmp-dir\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:10.262712 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.262692    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/279b3dd8-58be-43a1-9994-59fa434409f5-config-volume\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:10.267736 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.267715    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78b91dbb-d9e9-4524-8827-d74c8c0625d0-cert\") pod \"ingress-canary-6ldkm\" (UID: \"78b91dbb-d9e9-4524-8827-d74c8c0625d0\") " pod="openshift-ingress-canary/ingress-canary-6ldkm"
Nov 03 07:53:10.270708 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.270665    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp9jm\" (UniqueName: \"kubernetes.io/projected/279b3dd8-58be-43a1-9994-59fa434409f5-kube-api-access-lp9jm\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:10.270959 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.270893    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvnh8\" (UniqueName: \"kubernetes.io/projected/78b91dbb-d9e9-4524-8827-d74c8c0625d0-kube-api-access-kvnh8\") pod \"ingress-canary-6ldkm\" (UID: \"78b91dbb-d9e9-4524-8827-d74c8c0625d0\") " pod="openshift-ingress-canary/ingress-canary-6ldkm"
Nov 03 07:53:10.295134 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.295099    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:10.315235 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.315204    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk"
Nov 03 07:53:10.320524 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.320392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:10.327481 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.327304    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f"
Nov 03 07:53:10.332167 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.331930    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fshz5"
Nov 03 07:53:10.338000 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.337630    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zhh6g"
Nov 03 07:53:10.356370 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.356347    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6ldkm"
Nov 03 07:53:10.531594 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.530081    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-nwtkn"]
Nov 03 07:53:10.531594 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.531333    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kr2q8"]
Nov 03 07:53:10.534390 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:53:10.534352    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08817fd9_0879_4b77_a767_5f4c4ecf8d5b.slice/crio-2c89b4b61002542a42a86b9447902783481924f92225ac67a1eeccf1f1af5b61 WatchSource:0}: Error finding container 2c89b4b61002542a42a86b9447902783481924f92225ac67a1eeccf1f1af5b61: Status 404 returned error can't find the container with id 2c89b4b61002542a42a86b9447902783481924f92225ac67a1eeccf1f1af5b61
Nov 03 07:53:10.535516 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:53:10.535492    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf55612_a9f0_4fd5_ab00_ac2271acfb30.slice/crio-32284d065e9cfd2ff3d1d763bb2440817af436a19a096e3d43956a2a7429f7e8 WatchSource:0}: Error finding container 32284d065e9cfd2ff3d1d763bb2440817af436a19a096e3d43956a2a7429f7e8: Status 404 returned error can't find the container with id 32284d065e9cfd2ff3d1d763bb2440817af436a19a096e3d43956a2a7429f7e8
Nov 03 07:53:10.546709 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:53:10.546690    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b9100c_2539_4227_8f2a_c93660540664.slice/crio-a99672cb6ab93fd607c333290e8dacb4867c3021d76e584fa8756f7aa9385110 WatchSource:0}: Error finding container a99672cb6ab93fd607c333290e8dacb4867c3021d76e584fa8756f7aa9385110: Status 404 returned error can't find the container with id a99672cb6ab93fd607c333290e8dacb4867c3021d76e584fa8756f7aa9385110
Nov 03 07:53:10.547314 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.547270    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-v2z4k"]
Nov 03 07:53:10.626847 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.625948    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-crn8p"]
Nov 03 07:53:10.656625 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.656598    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk"]
Nov 03 07:53:10.666856 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.666272    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dtgx2\" (UID: \"dc162541-2ab5-4475-8856-850a7c5fd437\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:10.666856 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.666330    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:10.666856 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.666361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:10.666856 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.666403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-f68ks\" (UID: \"c383d3eb-10c7-4fe5-99b6-04c64472ef0a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:10.666856 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.666417    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 07:53:10.666856 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.666435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:10.666856 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.666474    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-tls podName:dc162541-2ab5-4475-8856-850a7c5fd437 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:11.666458385 +0000 UTC m=+31.534496099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dtgx2" (UID: "dc162541-2ab5-4475-8856-850a7c5fd437") : secret "samples-operator-tls" not found
Nov 03 07:53:10.666856 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.666516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/751205b0-8312-4ea6-920c-9517e19566f2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-grgjj\" (UID: \"751205b0-8312-4ea6-920c-9517e19566f2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grgjj"
Nov 03 07:53:10.666856 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.666535    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 07:53:10.666856 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.666549    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74774885c5-97jxz: secret "image-registry-tls" not found
Nov 03 07:53:10.666856 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.666593    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 07:53:10.666856 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.666632    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-metrics-certs podName:ce890601-127d-4995-9487-635e7aab04f8 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:11.66662061 +0000 UTC m=+31.534658322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-metrics-certs") pod "router-default-79698c9d55-m2lbh" (UID: "ce890601-127d-4995-9487-635e7aab04f8") : secret "router-metrics-certs-default" not found
Nov 03 07:53:10.666856 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.666655    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls podName:df604a27-2a06-42a0-bcbd-f7a10c1420c1 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:11.666639363 +0000 UTC m=+31.534677075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls") pod "image-registry-74774885c5-97jxz" (UID: "df604a27-2a06-42a0-bcbd-f7a10c1420c1") : secret "image-registry-tls" not found
Nov 03 07:53:10.666856 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.666549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-metrics-certs\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:10.666856 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.666670    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 07:53:10.666856 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.666737    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 07:53:10.666856 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.666747    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67ffc88546-z24dr: secret "image-registry-tls" not found
Nov 03 07:53:10.667691 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.666910    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 07:53:10.667691 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.667184    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/751205b0-8312-4ea6-920c-9517e19566f2-networking-console-plugin-cert podName:751205b0-8312-4ea6-920c-9517e19566f2 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:11.666685916 +0000 UTC m=+31.534723666 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/751205b0-8312-4ea6-920c-9517e19566f2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-grgjj" (UID: "751205b0-8312-4ea6-920c-9517e19566f2") : secret "networking-console-plugin-cert" not found
Nov 03 07:53:10.667691 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.667225    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls podName:119141fd-3c4f-4b94-9f18-6892986867e9 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:11.667206785 +0000 UTC m=+31.535244499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls") pod "image-registry-67ffc88546-z24dr" (UID: "119141fd-3c4f-4b94-9f18-6892986867e9") : secret "image-registry-tls" not found
Nov 03 07:53:10.667691 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.667250    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle podName:ce890601-127d-4995-9487-635e7aab04f8 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:11.667236713 +0000 UTC m=+31.535274680 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle") pod "router-default-79698c9d55-m2lbh" (UID: "ce890601-127d-4995-9487-635e7aab04f8") : configmap references non-existent config key: service-ca.crt
Nov 03 07:53:10.667691 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.667281    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-cluster-monitoring-operator-tls podName:c383d3eb-10c7-4fe5-99b6-04c64472ef0a nodeName:}" failed. No retries permitted until 2026-11-03 07:53:11.667269352 +0000 UTC m=+31.535307078 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-f68ks" (UID: "c383d3eb-10c7-4fe5-99b6-04c64472ef0a") : secret "cluster-monitoring-operator-tls" not found
Nov 03 07:53:10.674350 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.674327    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"]
Nov 03 07:53:10.686348 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:53:10.686319    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b91dbb_d9e9_4524_8827_d74c8c0625d0.slice/crio-0f773c5bcdf6c9660c4dab8c6b494a5750da8e38e2843d8662d052cdc2b59df0 WatchSource:0}: Error finding container 0f773c5bcdf6c9660c4dab8c6b494a5750da8e38e2843d8662d052cdc2b59df0: Status 404 returned error can't find the container with id 0f773c5bcdf6c9660c4dab8c6b494a5750da8e38e2843d8662d052cdc2b59df0
Nov 03 07:53:10.687520 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.687504    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6ldkm"]
Nov 03 07:53:10.687584 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.687540    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f"]
Nov 03 07:53:10.690079 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:53:10.690051    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58448f55_1670_41ce_8e35_182fe94f2d7e.slice/crio-20c64d2f5beeb6f50908c2f3e422d3393f3829ca8d4366021ef2f482b4794767 WatchSource:0}: Error finding container 20c64d2f5beeb6f50908c2f3e422d3393f3829ca8d4366021ef2f482b4794767: Status 404 returned error can't find the container with id 20c64d2f5beeb6f50908c2f3e422d3393f3829ca8d4366021ef2f482b4794767
Nov 03 07:53:10.698343 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.698324    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-fshz5"]
Nov 03 07:53:10.698953 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:53:10.698933    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e349708_a596_40ca_8724_6c2169cace8c.slice/crio-311618f3e5f892bfbf815f887a7944193e5a2914303767dc0afd89d90983ec68 WatchSource:0}: Error finding container 311618f3e5f892bfbf815f887a7944193e5a2914303767dc0afd89d90983ec68: Status 404 returned error can't find the container with id 311618f3e5f892bfbf815f887a7944193e5a2914303767dc0afd89d90983ec68
Nov 03 07:53:10.716785 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.716740    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zhh6g"]
Nov 03 07:53:10.719235 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:53:10.719215    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb721ff_442b_41b7_8287_220dbfbb41da.slice/crio-1ead0f5b21df840f477ce3be72c0d8eb6c637190655753167aa94fc5aa524f3e WatchSource:0}: Error finding container 1ead0f5b21df840f477ce3be72c0d8eb6c637190655753167aa94fc5aa524f3e: Status 404 returned error can't find the container with id 1ead0f5b21df840f477ce3be72c0d8eb6c637190655753167aa94fc5aa524f3e
Nov 03 07:53:10.730599 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.730582    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9j2f9"
Nov 03 07:53:10.730700 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.730693    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 07:53:10.731171 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.731157    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9j2f9"
Nov 03 07:53:10.768057 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.768033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/279b3dd8-58be-43a1-9994-59fa434409f5-metrics-tls\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:10.768176 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.768162    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 07:53:10.768212 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:10.768207    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/279b3dd8-58be-43a1-9994-59fa434409f5-metrics-tls podName:279b3dd8-58be-43a1-9994-59fa434409f5 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:11.768194366 +0000 UTC m=+31.636232065 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/279b3dd8-58be-43a1-9994-59fa434409f5-metrics-tls") pod "dns-default-46l5d" (UID: "279b3dd8-58be-43a1-9994-59fa434409f5") : secret "dns-default-metrics-tls" not found
Nov 03 07:53:10.793066 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.793047    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:10.793243 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.793219    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 07:53:10.808172 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.808152    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:10.954463 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.954412    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fshz5" event={"ID":"7e349708-a596-40ca-8724-6c2169cace8c","Type":"ContainerStarted","Data":"311618f3e5f892bfbf815f887a7944193e5a2914303767dc0afd89d90983ec68"}
Nov 03 07:53:10.955616 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.955595    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" event={"ID":"d83648de-81dd-4160-8fa9-06d36b38d9de","Type":"ContainerStarted","Data":"a56a831ed51419aecf02a89cbc7eb48de8f0e93848e8d73cf5975180502aaca1"}
Nov 03 07:53:10.956642 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.956615    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk" event={"ID":"e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3","Type":"ContainerStarted","Data":"414453077f111a09b67bb769a7b7b052ca00eeeba4a59290085399c8f9b2ceed"}
Nov 03 07:53:10.957553 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.957531    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-crn8p" event={"ID":"c620e360-6548-4b0c-ba9a-a9150cf9939c","Type":"ContainerStarted","Data":"b8e68337a90d4985c8d2a448ff2da30c8cd0ea586e404fe1610283e436ae38e7"}
Nov 03 07:53:10.958969 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.958949    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-v2z4k" event={"ID":"20b9100c-2539-4227-8f2a-c93660540664","Type":"ContainerStarted","Data":"b184ef8b73d1f3af213a65a14dd976dd0c4f67aa1fe68cdb082319fd62388c1f"}
Nov 03 07:53:10.959059 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.958984    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-v2z4k" event={"ID":"20b9100c-2539-4227-8f2a-c93660540664","Type":"ContainerStarted","Data":"a99672cb6ab93fd607c333290e8dacb4867c3021d76e584fa8756f7aa9385110"}
Nov 03 07:53:10.960082 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.960062    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zhh6g" event={"ID":"3cb721ff-442b-41b7-8287-220dbfbb41da","Type":"ContainerStarted","Data":"1ead0f5b21df840f477ce3be72c0d8eb6c637190655753167aa94fc5aa524f3e"}
Nov 03 07:53:10.961130 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.961112    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6ldkm" event={"ID":"78b91dbb-d9e9-4524-8827-d74c8c0625d0","Type":"ContainerStarted","Data":"0f773c5bcdf6c9660c4dab8c6b494a5750da8e38e2843d8662d052cdc2b59df0"}
Nov 03 07:53:10.962398 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.962376    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kr2q8" event={"ID":"edf55612-a9f0-4fd5-ab00-ac2271acfb30","Type":"ContainerStarted","Data":"c6d91a93fcc4302922e679e7e539619324f25a15fa468fe92da7c3b3da2b4b11"}
Nov 03 07:53:10.962483 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.962416    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kr2q8" event={"ID":"edf55612-a9f0-4fd5-ab00-ac2271acfb30","Type":"ContainerStarted","Data":"32284d065e9cfd2ff3d1d763bb2440817af436a19a096e3d43956a2a7429f7e8"}
Nov 03 07:53:10.963505 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.963484    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f" event={"ID":"58448f55-1670-41ce-8e35-182fe94f2d7e","Type":"ContainerStarted","Data":"20c64d2f5beeb6f50908c2f3e422d3393f3829ca8d4366021ef2f482b4794767"}
Nov 03 07:53:10.964806 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.964784    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nwtkn" event={"ID":"08817fd9-0879-4b77-a767-5f4c4ecf8d5b","Type":"ContainerStarted","Data":"9b0b7b77b1c67c29e83da9971e8504a1ac7986cdde79353cb76ace2a8130db4f"}
Nov 03 07:53:10.964869 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.964822    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-nwtkn" event={"ID":"08817fd9-0879-4b77-a767-5f4c4ecf8d5b","Type":"ContainerStarted","Data":"2c89b4b61002542a42a86b9447902783481924f92225ac67a1eeccf1f1af5b61"}
Nov 03 07:53:10.975402 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.975367    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-v2z4k" podStartSLOduration=425.975354757 podStartE2EDuration="7m5.975354757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:10.974470466 +0000 UTC m=+30.842508186" watchObservedRunningTime="2026-11-03 07:53:10.975354757 +0000 UTC m=+30.843392478"
Nov 03 07:53:10.992491 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:10.992457    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-nwtkn" podStartSLOduration=425.992445663 podStartE2EDuration="7m5.992445663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:10.991294579 +0000 UTC m=+30.859332300" watchObservedRunningTime="2026-11-03 07:53:10.992445663 +0000 UTC m=+30.860483411"
Nov 03 07:53:11.007900 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:11.007864    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-kr2q8" podStartSLOduration=426.007853722 podStartE2EDuration="7m6.007853722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:11.007653369 +0000 UTC m=+30.875691084" watchObservedRunningTime="2026-11-03 07:53:11.007853722 +0000 UTC m=+30.875891442"
Nov 03 07:53:11.678924 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:11.677793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dtgx2\" (UID: \"dc162541-2ab5-4475-8856-850a7c5fd437\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:11.678924 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:11.677856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:11.678924 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:11.677887    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:11.678924 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:11.677917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-f68ks\" (UID: \"c383d3eb-10c7-4fe5-99b6-04c64472ef0a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:11.678924 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:11.677949    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:11.678924 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:11.677993    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/751205b0-8312-4ea6-920c-9517e19566f2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-grgjj\" (UID: \"751205b0-8312-4ea6-920c-9517e19566f2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grgjj"
Nov 03 07:53:11.678924 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:11.678014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-metrics-certs\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:11.678924 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:11.678130    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 07:53:11.678924 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:11.678177    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-metrics-certs podName:ce890601-127d-4995-9487-635e7aab04f8 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:13.678165232 +0000 UTC m=+33.546202931 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-metrics-certs") pod "router-default-79698c9d55-m2lbh" (UID: "ce890601-127d-4995-9487-635e7aab04f8") : secret "router-metrics-certs-default" not found
Nov 03 07:53:11.678924 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:11.678500    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 07:53:11.678924 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:11.678540    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-tls podName:dc162541-2ab5-4475-8856-850a7c5fd437 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:13.678530618 +0000 UTC m=+33.546568329 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dtgx2" (UID: "dc162541-2ab5-4475-8856-850a7c5fd437") : secret "samples-operator-tls" not found
Nov 03 07:53:11.678924 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:11.678581    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 07:53:11.678924 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:11.678588    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67ffc88546-z24dr: secret "image-registry-tls" not found
Nov 03 07:53:11.678924 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:11.678609    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls podName:119141fd-3c4f-4b94-9f18-6892986867e9 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:13.678601955 +0000 UTC m=+33.546639667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls") pod "image-registry-67ffc88546-z24dr" (UID: "119141fd-3c4f-4b94-9f18-6892986867e9") : secret "image-registry-tls" not found
Nov 03 07:53:11.678924 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:11.678649    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle podName:ce890601-127d-4995-9487-635e7aab04f8 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:13.678642661 +0000 UTC m=+33.546680374 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle") pod "router-default-79698c9d55-m2lbh" (UID: "ce890601-127d-4995-9487-635e7aab04f8") : configmap references non-existent config key: service-ca.crt
Nov 03 07:53:11.678924 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:11.678683    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 07:53:11.679804 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:11.678710    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-cluster-monitoring-operator-tls podName:c383d3eb-10c7-4fe5-99b6-04c64472ef0a nodeName:}" failed. No retries permitted until 2026-11-03 07:53:13.678701869 +0000 UTC m=+33.546739584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-f68ks" (UID: "c383d3eb-10c7-4fe5-99b6-04c64472ef0a") : secret "cluster-monitoring-operator-tls" not found
Nov 03 07:53:11.679804 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:11.678750    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 07:53:11.679804 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:11.678777    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74774885c5-97jxz: secret "image-registry-tls" not found
Nov 03 07:53:11.679804 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:11.678806    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls podName:df604a27-2a06-42a0-bcbd-f7a10c1420c1 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:13.678796349 +0000 UTC m=+33.546834062 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls") pod "image-registry-74774885c5-97jxz" (UID: "df604a27-2a06-42a0-bcbd-f7a10c1420c1") : secret "image-registry-tls" not found
Nov 03 07:53:11.679804 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:11.678854    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 07:53:11.679804 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:11.678882    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/751205b0-8312-4ea6-920c-9517e19566f2-networking-console-plugin-cert podName:751205b0-8312-4ea6-920c-9517e19566f2 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:13.678873638 +0000 UTC m=+33.546911350 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/751205b0-8312-4ea6-920c-9517e19566f2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-grgjj" (UID: "751205b0-8312-4ea6-920c-9517e19566f2") : secret "networking-console-plugin-cert" not found
Nov 03 07:53:11.683246 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:11.681663    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:53:11.683246 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:11.682118    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:53:11.683246 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:11.682468    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:53:11.686543 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:11.686378    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 07:53:11.686543 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:11.686407    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 07:53:11.686543 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:11.686452    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5b8js\""
Nov 03 07:53:11.686543 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:11.686481    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hq49v\""
Nov 03 07:53:11.686543 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:11.686523    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dl4bp\""
Nov 03 07:53:11.778654 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:11.778623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/279b3dd8-58be-43a1-9994-59fa434409f5-metrics-tls\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:11.778885 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:11.778868    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 07:53:11.778945 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:11.778932    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/279b3dd8-58be-43a1-9994-59fa434409f5-metrics-tls podName:279b3dd8-58be-43a1-9994-59fa434409f5 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:13.778913524 +0000 UTC m=+33.646951238 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/279b3dd8-58be-43a1-9994-59fa434409f5-metrics-tls") pod "dns-default-46l5d" (UID: "279b3dd8-58be-43a1-9994-59fa434409f5") : secret "dns-default-metrics-tls" not found
Nov 03 07:53:13.699128 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:13.699099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:13.699438 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:13.699172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/751205b0-8312-4ea6-920c-9517e19566f2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-grgjj\" (UID: \"751205b0-8312-4ea6-920c-9517e19566f2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grgjj"
Nov 03 07:53:13.699438 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:13.699210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-metrics-certs\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:13.699438 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:13.699307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dtgx2\" (UID: \"dc162541-2ab5-4475-8856-850a7c5fd437\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:13.699438 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:13.699348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:13.699438 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:13.699378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:13.699438 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:13.699406    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-f68ks\" (UID: \"c383d3eb-10c7-4fe5-99b6-04c64472ef0a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:13.699634 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:13.699550    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 07:53:13.699634 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:13.699615    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-cluster-monitoring-operator-tls podName:c383d3eb-10c7-4fe5-99b6-04c64472ef0a nodeName:}" failed. No retries permitted until 2026-11-03 07:53:17.699596855 +0000 UTC m=+37.567634568 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-f68ks" (UID: "c383d3eb-10c7-4fe5-99b6-04c64472ef0a") : secret "cluster-monitoring-operator-tls" not found
Nov 03 07:53:13.700040 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:13.700022    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 07:53:13.700110 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:13.700045    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74774885c5-97jxz: secret "image-registry-tls" not found
Nov 03 07:53:13.700110 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:13.700092    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls podName:df604a27-2a06-42a0-bcbd-f7a10c1420c1 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:17.700075985 +0000 UTC m=+37.568113699 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls") pod "image-registry-74774885c5-97jxz" (UID: "df604a27-2a06-42a0-bcbd-f7a10c1420c1") : secret "image-registry-tls" not found
Nov 03 07:53:13.700201 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:13.700150    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 07:53:13.700201 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:13.700185    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/751205b0-8312-4ea6-920c-9517e19566f2-networking-console-plugin-cert podName:751205b0-8312-4ea6-920c-9517e19566f2 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:17.700173047 +0000 UTC m=+37.568210761 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/751205b0-8312-4ea6-920c-9517e19566f2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-grgjj" (UID: "751205b0-8312-4ea6-920c-9517e19566f2") : secret "networking-console-plugin-cert" not found
Nov 03 07:53:13.700262 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:13.700226    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 07:53:13.700262 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:13.700247    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-metrics-certs podName:ce890601-127d-4995-9487-635e7aab04f8 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:17.70024096 +0000 UTC m=+37.568278672 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-metrics-certs") pod "router-default-79698c9d55-m2lbh" (UID: "ce890601-127d-4995-9487-635e7aab04f8") : secret "router-metrics-certs-default" not found
Nov 03 07:53:13.700323 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:13.700278    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 07:53:13.700323 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:13.700296    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-tls podName:dc162541-2ab5-4475-8856-850a7c5fd437 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:17.700291103 +0000 UTC m=+37.568328815 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dtgx2" (UID: "dc162541-2ab5-4475-8856-850a7c5fd437") : secret "samples-operator-tls" not found
Nov 03 07:53:13.700401 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:13.700356    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 07:53:13.700401 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:13.700365    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67ffc88546-z24dr: secret "image-registry-tls" not found
Nov 03 07:53:13.700401 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:13.700393    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls podName:119141fd-3c4f-4b94-9f18-6892986867e9 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:17.700383297 +0000 UTC m=+37.568421010 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls") pod "image-registry-67ffc88546-z24dr" (UID: "119141fd-3c4f-4b94-9f18-6892986867e9") : secret "image-registry-tls" not found
Nov 03 07:53:13.700486 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:13.700435    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle podName:ce890601-127d-4995-9487-635e7aab04f8 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:17.700429118 +0000 UTC m=+37.568466830 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle") pod "router-default-79698c9d55-m2lbh" (UID: "ce890601-127d-4995-9487-635e7aab04f8") : configmap references non-existent config key: service-ca.crt
Nov 03 07:53:13.800982 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:13.800945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/279b3dd8-58be-43a1-9994-59fa434409f5-metrics-tls\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:13.801203 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:13.801077    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 07:53:13.801271 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:13.801219    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/279b3dd8-58be-43a1-9994-59fa434409f5-metrics-tls podName:279b3dd8-58be-43a1-9994-59fa434409f5 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:17.801197477 +0000 UTC m=+37.669235176 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/279b3dd8-58be-43a1-9994-59fa434409f5-metrics-tls") pod "dns-default-46l5d" (UID: "279b3dd8-58be-43a1-9994-59fa434409f5") : secret "dns-default-metrics-tls" not found
Nov 03 07:53:13.981000 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:13.980339    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zhh6g" event={"ID":"3cb721ff-442b-41b7-8287-220dbfbb41da","Type":"ContainerStarted","Data":"cce25c73e01c1f7bab5cc0b02b1dd1064529fd8f5d3b799846425f3124bac362"}
Nov 03 07:53:14.309138 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:14.309000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:53:14.311711 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:14.311676    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:53:14.409975 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:14.409934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs\") pod \"network-metrics-daemon-x7cmt\" (UID: \"25a7353f-76ad-4ac0-92b8-186377723478\") " pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:53:14.410129 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:14.410112    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 07:53:14.410180 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:14.410130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k67j\" (UniqueName: \"kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j\") pod \"network-check-target-c6bms\" (UID: \"3a9e7292-b781-4a27-8a48-94f0a8843fc2\") " pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:53:14.410214 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:14.410186    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs podName:25a7353f-76ad-4ac0-92b8-186377723478 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:46.410167775 +0000 UTC m=+66.278205492 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs") pod "network-metrics-daemon-x7cmt" (UID: "25a7353f-76ad-4ac0-92b8-186377723478") : secret "metrics-daemon-secret" not found
Nov 03 07:53:14.413274 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:14.413248    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k67j\" (UniqueName: \"kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j\") pod \"network-check-target-c6bms\" (UID: \"3a9e7292-b781-4a27-8a48-94f0a8843fc2\") " pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:53:14.464634 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:14.464458    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:53:14.711268 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:14.711227    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:53:15.160565 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.160391    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zhh6g" podStartSLOduration=403.877832263 podStartE2EDuration="6m46.160375089s" totalImagesPullingTime="2.282542826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:13.99745 +0000 UTC m=+33.865487720" watchObservedRunningTime="2026-11-03 07:53:15.160375089 +0000 UTC m=+35.028412815"
Nov 03 07:53:15.160565 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.160511    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c6bms"]
Nov 03 07:53:15.177017 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.174150    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ljbh2"]
Nov 03 07:53:15.557874 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.557799    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wd497"]
Nov 03 07:53:15.577383 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.577351    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-l5f6q-fpwdg"]
Nov 03 07:53:15.577537 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.577462    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:15.582042 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.582018    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5spf6\""
Nov 03 07:53:15.583137 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.582634    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 07:53:15.583137 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.582700    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 07:53:15.596213 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.596192    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wd497"]
Nov 03 07:53:15.596302 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.596231    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-l5f6q-fpwdg"]
Nov 03 07:53:15.596372 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.596307    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:15.622909 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.622886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klntp\" (UniqueName: \"kubernetes.io/projected/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-kube-api-access-klntp\") pod \"periodic-gathering-l5f6q-fpwdg\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") " pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:15.623025 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.622927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6731c200-9eda-44f2-aa13-2b4a8fbda85e-crio-socket\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:15.623025 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.622954    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:15.623151 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.623126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6731c200-9eda-44f2-aa13-2b4a8fbda85e-data-volume\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:15.623266 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.623164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-archives-path\") pod \"periodic-gathering-l5f6q-fpwdg\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") " pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:15.623266 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.623180    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-trusted-ca-bundle\") pod \"periodic-gathering-l5f6q-fpwdg\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") " pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:15.623266 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.623212    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48c5j\" (UniqueName: \"kubernetes.io/projected/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-api-access-48c5j\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:15.623266 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.623238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-service-ca-bundle\") pod \"periodic-gathering-l5f6q-fpwdg\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") " pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:15.623464 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.623290    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:15.682195 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.682156    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rmlwc\""
Nov 03 07:53:15.724025 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.723997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klntp\" (UniqueName: \"kubernetes.io/projected/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-kube-api-access-klntp\") pod \"periodic-gathering-l5f6q-fpwdg\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") " pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:15.724025 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.724028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6731c200-9eda-44f2-aa13-2b4a8fbda85e-crio-socket\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:15.724460 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.724044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:15.724460 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.724132    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6731c200-9eda-44f2-aa13-2b4a8fbda85e-crio-socket\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:15.724460 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.724158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6731c200-9eda-44f2-aa13-2b4a8fbda85e-data-volume\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:15.724460 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.724199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-archives-path\") pod \"periodic-gathering-l5f6q-fpwdg\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") " pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:15.724460 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.724223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-trusted-ca-bundle\") pod \"periodic-gathering-l5f6q-fpwdg\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") " pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:15.724460 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:15.724232    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 07:53:15.724460 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.724265    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48c5j\" (UniqueName: \"kubernetes.io/projected/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-api-access-48c5j\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:15.724460 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:15.724285    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls podName:6731c200-9eda-44f2-aa13-2b4a8fbda85e nodeName:}" failed. No retries permitted until 2026-11-03 07:53:16.224267334 +0000 UTC m=+36.092305045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wd497" (UID: "6731c200-9eda-44f2-aa13-2b4a8fbda85e") : secret "insights-runtime-extractor-tls" not found
Nov 03 07:53:15.724460 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.724325    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-service-ca-bundle\") pod \"periodic-gathering-l5f6q-fpwdg\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") " pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:15.724460 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.724397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:15.725044 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:15.724479    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 07:53:15.725044 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.724513    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6731c200-9eda-44f2-aa13-2b4a8fbda85e-data-volume\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:15.725044 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:15.724546    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-rbac-proxy-cm podName:6731c200-9eda-44f2-aa13-2b4a8fbda85e nodeName:}" failed. No retries permitted until 2026-11-03 07:53:16.224530645 +0000 UTC m=+36.092568345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wd497" (UID: "6731c200-9eda-44f2-aa13-2b4a8fbda85e") : configmap "kube-rbac-proxy" not found
Nov 03 07:53:15.725044 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.724641    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-archives-path\") pod \"periodic-gathering-l5f6q-fpwdg\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") " pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:15.725044 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.724997    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-service-ca-bundle\") pod \"periodic-gathering-l5f6q-fpwdg\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") " pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:15.725689 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.725665    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-trusted-ca-bundle\") pod \"periodic-gathering-l5f6q-fpwdg\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") " pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:15.734374 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.734355    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48c5j\" (UniqueName: \"kubernetes.io/projected/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-api-access-48c5j\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:15.737647 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.737627    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klntp\" (UniqueName: \"kubernetes.io/projected/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-kube-api-access-klntp\") pod \"periodic-gathering-l5f6q-fpwdg\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") " pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:15.953511 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.953483    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:15.989143 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.989117    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f" event={"ID":"58448f55-1670-41ce-8e35-182fe94f2d7e","Type":"ContainerStarted","Data":"17f816370e04394a061fee875159364543025fd7ac60dca582b06ada5f984d25"}
Nov 03 07:53:15.992596 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.992568    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ljbh2" event={"ID":"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df","Type":"ContainerStarted","Data":"bac266740cb2cba0a7f336dd16bb29f18af761b13f88a307e6df1e5f7d966c63"}
Nov 03 07:53:15.993983 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.993965    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/0.log"
Nov 03 07:53:15.994077 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.993996    2579 generic.go:356] "Generic (PLEG): container finished" podID="d83648de-81dd-4160-8fa9-06d36b38d9de" containerID="7aa5bbf018c5c87f94df8f87b4992d055c8cd7df3e94be913efd00ebc8201046" exitCode=255
Nov 03 07:53:15.994077 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.994056    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" event={"ID":"d83648de-81dd-4160-8fa9-06d36b38d9de","Type":"ContainerDied","Data":"7aa5bbf018c5c87f94df8f87b4992d055c8cd7df3e94be913efd00ebc8201046"}
Nov 03 07:53:15.994267 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.994253    2579 scope.go:122] "RemoveContainer" containerID="7aa5bbf018c5c87f94df8f87b4992d055c8cd7df3e94be913efd00ebc8201046"
Nov 03 07:53:15.995176 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.995157    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c6bms" event={"ID":"3a9e7292-b781-4a27-8a48-94f0a8843fc2","Type":"ContainerStarted","Data":"d1d4139834fa8ce4e8bab7ae6dbff1ed5ea071f1d006baa6775ed8c3b53552a7"}
Nov 03 07:53:15.996543 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:15.996523    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-crn8p" event={"ID":"c620e360-6548-4b0c-ba9a-a9150cf9939c","Type":"ContainerStarted","Data":"c9d39725eee87abc02081ab62a587cea0577e4237fa7bad287fbcc259af7aef1"}
Nov 03 07:53:16.005777 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.005717    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f" podStartSLOduration=402.385927387 podStartE2EDuration="6m47.005705372s" totalImagesPullingTime="4.619777985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:16.004077333 +0000 UTC m=+35.872115055" watchObservedRunningTime="2026-11-03 07:53:16.005705372 +0000 UTC m=+35.873743094"
Nov 03 07:53:16.020472 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.020429    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-crn8p" podStartSLOduration=402.769171658 podStartE2EDuration="6m47.020417036s" totalImagesPullingTime="4.251245378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:16.019094414 +0000 UTC m=+35.887132135" watchObservedRunningTime="2026-11-03 07:53:16.020417036 +0000 UTC m=+35.888454756"
Nov 03 07:53:16.228685 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.228615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:16.228825 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:16.228785    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 07:53:16.228863 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:16.228854    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls podName:6731c200-9eda-44f2-aa13-2b4a8fbda85e nodeName:}" failed. No retries permitted until 2026-11-03 07:53:17.228837456 +0000 UTC m=+37.096875167 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wd497" (UID: "6731c200-9eda-44f2-aa13-2b4a8fbda85e") : secret "insights-runtime-extractor-tls" not found
Nov 03 07:53:16.228916 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.228900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:16.228998 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:16.228986    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 07:53:16.229041 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:16.229031    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-rbac-proxy-cm podName:6731c200-9eda-44f2-aa13-2b4a8fbda85e nodeName:}" failed. No retries permitted until 2026-11-03 07:53:17.229017795 +0000 UTC m=+37.097055494 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wd497" (UID: "6731c200-9eda-44f2-aa13-2b4a8fbda85e") : configmap "kube-rbac-proxy" not found
Nov 03 07:53:16.531219 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.531152    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-cnskv"]
Nov 03 07:53:16.564062 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.564011    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-cnskv"]
Nov 03 07:53:16.564162 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.564134    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-cnskv"
Nov 03 07:53:16.568524 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.568411    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 07:53:16.568524 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.568460    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 07:53:16.568524 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.568463    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 07:53:16.569734 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.569716    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-tpfjw\""
Nov 03 07:53:16.569821 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.569726    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 07:53:16.636083 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.635859    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22cnl\" (UniqueName: \"kubernetes.io/projected/96cfe445-92c7-4bfe-bafd-8943ce48cee8-kube-api-access-22cnl\") pod \"service-ca-5fffc97bdf-cnskv\" (UID: \"96cfe445-92c7-4bfe-bafd-8943ce48cee8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cnskv"
Nov 03 07:53:16.636083 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.635918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/96cfe445-92c7-4bfe-bafd-8943ce48cee8-signing-key\") pod \"service-ca-5fffc97bdf-cnskv\" (UID: \"96cfe445-92c7-4bfe-bafd-8943ce48cee8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cnskv"
Nov 03 07:53:16.636083 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.635985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/96cfe445-92c7-4bfe-bafd-8943ce48cee8-signing-cabundle\") pod \"service-ca-5fffc97bdf-cnskv\" (UID: \"96cfe445-92c7-4bfe-bafd-8943ce48cee8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cnskv"
Nov 03 07:53:16.736812 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.736729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/96cfe445-92c7-4bfe-bafd-8943ce48cee8-signing-cabundle\") pod \"service-ca-5fffc97bdf-cnskv\" (UID: \"96cfe445-92c7-4bfe-bafd-8943ce48cee8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cnskv"
Nov 03 07:53:16.737201 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.736846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22cnl\" (UniqueName: \"kubernetes.io/projected/96cfe445-92c7-4bfe-bafd-8943ce48cee8-kube-api-access-22cnl\") pod \"service-ca-5fffc97bdf-cnskv\" (UID: \"96cfe445-92c7-4bfe-bafd-8943ce48cee8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cnskv"
Nov 03 07:53:16.737201 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.736933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/96cfe445-92c7-4bfe-bafd-8943ce48cee8-signing-key\") pod \"service-ca-5fffc97bdf-cnskv\" (UID: \"96cfe445-92c7-4bfe-bafd-8943ce48cee8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cnskv"
Nov 03 07:53:16.738159 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.738113    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/96cfe445-92c7-4bfe-bafd-8943ce48cee8-signing-cabundle\") pod \"service-ca-5fffc97bdf-cnskv\" (UID: \"96cfe445-92c7-4bfe-bafd-8943ce48cee8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cnskv"
Nov 03 07:53:16.739862 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.739836    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/96cfe445-92c7-4bfe-bafd-8943ce48cee8-signing-key\") pod \"service-ca-5fffc97bdf-cnskv\" (UID: \"96cfe445-92c7-4bfe-bafd-8943ce48cee8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cnskv"
Nov 03 07:53:16.749779 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.749741    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-l5f6q-fpwdg"]
Nov 03 07:53:16.753298 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.753255    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22cnl\" (UniqueName: \"kubernetes.io/projected/96cfe445-92c7-4bfe-bafd-8943ce48cee8-kube-api-access-22cnl\") pod \"service-ca-5fffc97bdf-cnskv\" (UID: \"96cfe445-92c7-4bfe-bafd-8943ce48cee8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cnskv"
Nov 03 07:53:16.767812 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:53:16.767787    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e53c53_b31d_43f7_b7b2_2c90a07dcf09.slice/crio-9c0e481f1c0afebf976b20aa412a47bc61e96b6a980d3ca65fdde537b1ff35ce WatchSource:0}: Error finding container 9c0e481f1c0afebf976b20aa412a47bc61e96b6a980d3ca65fdde537b1ff35ce: Status 404 returned error can't find the container with id 9c0e481f1c0afebf976b20aa412a47bc61e96b6a980d3ca65fdde537b1ff35ce
Nov 03 07:53:16.913465 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:16.913438    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-cnskv"
Nov 03 07:53:17.008492 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.007992    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fshz5" event={"ID":"7e349708-a596-40ca-8724-6c2169cace8c","Type":"ContainerStarted","Data":"f5181d7d31aa65e2d2d0ca68e326af0bb83f037866ee987444d0cd41a7d5ccb7"}
Nov 03 07:53:17.013172 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.013120    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/1.log"
Nov 03 07:53:17.014932 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.013854    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/0.log"
Nov 03 07:53:17.014932 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.013890    2579 generic.go:356] "Generic (PLEG): container finished" podID="d83648de-81dd-4160-8fa9-06d36b38d9de" containerID="b17a031689eca024fee8f71239c507cd7dc10ca5e0147ff1e9b4d8440b1d8575" exitCode=255
Nov 03 07:53:17.014932 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.013961    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" event={"ID":"d83648de-81dd-4160-8fa9-06d36b38d9de","Type":"ContainerDied","Data":"b17a031689eca024fee8f71239c507cd7dc10ca5e0147ff1e9b4d8440b1d8575"}
Nov 03 07:53:17.014932 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.014005    2579 scope.go:122] "RemoveContainer" containerID="7aa5bbf018c5c87f94df8f87b4992d055c8cd7df3e94be913efd00ebc8201046"
Nov 03 07:53:17.014932 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.014380    2579 scope.go:122] "RemoveContainer" containerID="b17a031689eca024fee8f71239c507cd7dc10ca5e0147ff1e9b4d8440b1d8575"
Nov 03 07:53:17.014932 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.014594    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5qpnb_openshift-console-operator(d83648de-81dd-4160-8fa9-06d36b38d9de)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" podUID="d83648de-81dd-4160-8fa9-06d36b38d9de"
Nov 03 07:53:17.022241 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.021672    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c6bms" event={"ID":"3a9e7292-b781-4a27-8a48-94f0a8843fc2","Type":"ContainerStarted","Data":"4504c8fad9213c14b14229af94a737c6633fb61a9dd71db27c1a055ff77cd30c"}
Nov 03 07:53:17.022241 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.022177    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:53:17.024153 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.024096    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk" event={"ID":"e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3","Type":"ContainerStarted","Data":"b4a05aaeeb5ccdf147f3672d11c6d2a259fea578dd06392020786c013e31fff8"}
Nov 03 07:53:17.027126 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.026463    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-l5f6q-fpwdg" event={"ID":"66e53c53-b31d-43f7-b7b2-2c90a07dcf09","Type":"ContainerStarted","Data":"de2e064a851278745f4321cebf8e8272e783ea5b321a5e4ff3ccc6634c7f30e3"}
Nov 03 07:53:17.027126 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.026502    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-l5f6q-fpwdg" event={"ID":"66e53c53-b31d-43f7-b7b2-2c90a07dcf09","Type":"ContainerStarted","Data":"9c0e481f1c0afebf976b20aa412a47bc61e96b6a980d3ca65fdde537b1ff35ce"}
Nov 03 07:53:17.027126 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.026579    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fshz5" podStartSLOduration=396.265684953 podStartE2EDuration="6m42.026566246s" totalImagesPullingTime="5.760881293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:46:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:17.024710753 +0000 UTC m=+36.892748476" watchObservedRunningTime="2026-11-03 07:53:17.026566246 +0000 UTC m=+36.894603968"
Nov 03 07:53:17.029792 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.029768    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6ldkm" event={"ID":"78b91dbb-d9e9-4524-8827-d74c8c0625d0","Type":"ContainerStarted","Data":"f86521bea4ddaf095807e1f05e1cc73fb9879e8697f54835a435678bec78ac96"}
Nov 03 07:53:17.047732 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.047325    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-c6bms" podStartSLOduration=35.880896918 podStartE2EDuration="37.047312164s" totalImagesPullingTime="1.166415246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:17.045886155 +0000 UTC m=+36.913923877" watchObservedRunningTime="2026-11-03 07:53:17.047312164 +0000 UTC m=+36.915349885"
Nov 03 07:53:17.079130 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.078728    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk" podStartSLOduration=402.276162215 podStartE2EDuration="6m48.078715014s" totalImagesPullingTime="5.802552799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:17.078173695 +0000 UTC m=+36.946211418" watchObservedRunningTime="2026-11-03 07:53:17.078715014 +0000 UTC m=+36.946752732"
Nov 03 07:53:17.094971 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.094926    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6ldkm" podStartSLOduration=1.3243202250000001 podStartE2EDuration="7.094909796s" totalImagesPullingTime="5.770589571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:17.093232975 +0000 UTC m=+36.961270698" watchObservedRunningTime="2026-11-03 07:53:17.094909796 +0000 UTC m=+36.962947517"
Nov 03 07:53:17.111967 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.111929    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-l5f6q-fpwdg" podStartSLOduration=2.111914198 podStartE2EDuration="2.111914198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:17.110955794 +0000 UTC m=+36.978993514" watchObservedRunningTime="2026-11-03 07:53:17.111914198 +0000 UTC m=+36.979951922"
Nov 03 07:53:17.218223 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.218189    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-cnskv"]
Nov 03 07:53:17.221541 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:53:17.221514    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cfe445_92c7_4bfe_bafd_8943ce48cee8.slice/crio-33f8ce7f9bdc0222440db4a7bb237998b388778284b0ce351815314c1edcfd93 WatchSource:0}: Error finding container 33f8ce7f9bdc0222440db4a7bb237998b388778284b0ce351815314c1edcfd93: Status 404 returned error can't find the container with id 33f8ce7f9bdc0222440db4a7bb237998b388778284b0ce351815314c1edcfd93
Nov 03 07:53:17.242443 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.241647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:17.242443 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.241876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:17.242443 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.241970    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 07:53:17.242443 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.242021    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-rbac-proxy-cm podName:6731c200-9eda-44f2-aa13-2b4a8fbda85e nodeName:}" failed. No retries permitted until 2026-11-03 07:53:19.242005559 +0000 UTC m=+39.110043273 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wd497" (UID: "6731c200-9eda-44f2-aa13-2b4a8fbda85e") : configmap "kube-rbac-proxy" not found
Nov 03 07:53:17.242443 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.242366    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 07:53:17.242443 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.242410    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls podName:6731c200-9eda-44f2-aa13-2b4a8fbda85e nodeName:}" failed. No retries permitted until 2026-11-03 07:53:19.242396605 +0000 UTC m=+39.110434321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wd497" (UID: "6731c200-9eda-44f2-aa13-2b4a8fbda85e") : secret "insights-runtime-extractor-tls" not found
Nov 03 07:53:17.375648 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.375616    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-d7zfg"]
Nov 03 07:53:17.395328 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.394915    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fhz8r"]
Nov 03 07:53:17.395525 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.395508    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-d7zfg"
Nov 03 07:53:17.402209 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.402182    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vbzhs\""
Nov 03 07:53:17.405273 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.404724    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 07:53:17.405273 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.404846    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 07:53:17.413490 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.413145    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fhz8r"
Nov 03 07:53:17.413490 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.413179    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-d7zfg"]
Nov 03 07:53:17.413490 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.413211    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fhz8r"]
Nov 03 07:53:17.443460 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.443429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rbld\" (UniqueName: \"kubernetes.io/projected/fc09b315-54d3-44f0-b6ea-8028bd3b212b-kube-api-access-9rbld\") pod \"migrator-66df5d496b-d7zfg\" (UID: \"fc09b315-54d3-44f0-b6ea-8028bd3b212b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-d7zfg"
Nov 03 07:53:17.443646 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.443622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kk6l\" (UniqueName: \"kubernetes.io/projected/1778099a-ebc0-47ba-98d0-2b0e5726e7d2-kube-api-access-2kk6l\") pod \"migrator-66df5d496b-fhz8r\" (UID: \"1778099a-ebc0-47ba-98d0-2b0e5726e7d2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fhz8r"
Nov 03 07:53:17.544556 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.544466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kk6l\" (UniqueName: \"kubernetes.io/projected/1778099a-ebc0-47ba-98d0-2b0e5726e7d2-kube-api-access-2kk6l\") pod \"migrator-66df5d496b-fhz8r\" (UID: \"1778099a-ebc0-47ba-98d0-2b0e5726e7d2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fhz8r"
Nov 03 07:53:17.544697 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.544567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rbld\" (UniqueName: \"kubernetes.io/projected/fc09b315-54d3-44f0-b6ea-8028bd3b212b-kube-api-access-9rbld\") pod \"migrator-66df5d496b-d7zfg\" (UID: \"fc09b315-54d3-44f0-b6ea-8028bd3b212b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-d7zfg"
Nov 03 07:53:17.555327 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.555299    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rbld\" (UniqueName: \"kubernetes.io/projected/fc09b315-54d3-44f0-b6ea-8028bd3b212b-kube-api-access-9rbld\") pod \"migrator-66df5d496b-d7zfg\" (UID: \"fc09b315-54d3-44f0-b6ea-8028bd3b212b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-d7zfg"
Nov 03 07:53:17.555446 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.555331    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kk6l\" (UniqueName: \"kubernetes.io/projected/1778099a-ebc0-47ba-98d0-2b0e5726e7d2-kube-api-access-2kk6l\") pod \"migrator-66df5d496b-fhz8r\" (UID: \"1778099a-ebc0-47ba-98d0-2b0e5726e7d2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fhz8r"
Nov 03 07:53:17.743871 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.743838    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-d7zfg"
Nov 03 07:53:17.746300 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.746271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dtgx2\" (UID: \"dc162541-2ab5-4475-8856-850a7c5fd437\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:17.746431 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.746330    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:17.746431 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.746359    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:17.746561 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.746444    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Nov 03 07:53:17.746561 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.746460    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 07:53:17.746561 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.746480    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67ffc88546-z24dr: secret "image-registry-tls" not found
Nov 03 07:53:17.746561 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.746525    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-tls podName:dc162541-2ab5-4475-8856-850a7c5fd437 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:25.746494856 +0000 UTC m=+45.614532561 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-dtgx2" (UID: "dc162541-2ab5-4475-8856-850a7c5fd437") : secret "samples-operator-tls" not found
Nov 03 07:53:17.746561 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.746544    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle podName:ce890601-127d-4995-9487-635e7aab04f8 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:25.746534876 +0000 UTC m=+45.614572588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle") pod "router-default-79698c9d55-m2lbh" (UID: "ce890601-127d-4995-9487-635e7aab04f8") : configmap references non-existent config key: service-ca.crt
Nov 03 07:53:17.746561 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.746560    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls podName:119141fd-3c4f-4b94-9f18-6892986867e9 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:25.746550489 +0000 UTC m=+45.614588197 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls") pod "image-registry-67ffc88546-z24dr" (UID: "119141fd-3c4f-4b94-9f18-6892986867e9") : secret "image-registry-tls" not found
Nov 03 07:53:17.746870 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.746583    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-f68ks\" (UID: \"c383d3eb-10c7-4fe5-99b6-04c64472ef0a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:17.746870 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.746623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:17.746870 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.746681    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 07:53:17.746870 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.746689    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74774885c5-97jxz: secret "image-registry-tls" not found
Nov 03 07:53:17.746870 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.746689    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Nov 03 07:53:17.746870 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.746711    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls podName:df604a27-2a06-42a0-bcbd-f7a10c1420c1 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:25.746705004 +0000 UTC m=+45.614742704 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls") pod "image-registry-74774885c5-97jxz" (UID: "df604a27-2a06-42a0-bcbd-f7a10c1420c1") : secret "image-registry-tls" not found
Nov 03 07:53:17.746870 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.746781    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/751205b0-8312-4ea6-920c-9517e19566f2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-grgjj\" (UID: \"751205b0-8312-4ea6-920c-9517e19566f2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grgjj"
Nov 03 07:53:17.746870 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.746793    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-cluster-monitoring-operator-tls podName:c383d3eb-10c7-4fe5-99b6-04c64472ef0a nodeName:}" failed. No retries permitted until 2026-11-03 07:53:25.746781998 +0000 UTC m=+45.614819700 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-f68ks" (UID: "c383d3eb-10c7-4fe5-99b6-04c64472ef0a") : secret "cluster-monitoring-operator-tls" not found
Nov 03 07:53:17.746870 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.746815    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-metrics-certs\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:17.746870 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.746835    2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Nov 03 07:53:17.746870 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.746867    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/751205b0-8312-4ea6-920c-9517e19566f2-networking-console-plugin-cert podName:751205b0-8312-4ea6-920c-9517e19566f2 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:25.746853018 +0000 UTC m=+45.614890731 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/751205b0-8312-4ea6-920c-9517e19566f2-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-grgjj" (UID: "751205b0-8312-4ea6-920c-9517e19566f2") : secret "networking-console-plugin-cert" not found
Nov 03 07:53:17.747245 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.746896    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Nov 03 07:53:17.747245 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.746933    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-metrics-certs podName:ce890601-127d-4995-9487-635e7aab04f8 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:25.746923995 +0000 UTC m=+45.614961694 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-metrics-certs") pod "router-default-79698c9d55-m2lbh" (UID: "ce890601-127d-4995-9487-635e7aab04f8") : secret "router-metrics-certs-default" not found
Nov 03 07:53:17.763556 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.763537    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fhz8r"
Nov 03 07:53:17.848089 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:17.848062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/279b3dd8-58be-43a1-9994-59fa434409f5-metrics-tls\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:17.848236 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.848193    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 07:53:17.848291 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:17.848242    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/279b3dd8-58be-43a1-9994-59fa434409f5-metrics-tls podName:279b3dd8-58be-43a1-9994-59fa434409f5 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:25.848228474 +0000 UTC m=+45.716266187 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/279b3dd8-58be-43a1-9994-59fa434409f5-metrics-tls") pod "dns-default-46l5d" (UID: "279b3dd8-58be-43a1-9994-59fa434409f5") : secret "dns-default-metrics-tls" not found
Nov 03 07:53:18.048513 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:18.048487    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-d7zfg"]
Nov 03 07:53:18.054721 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:18.054666    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-d7zfg" event={"ID":"fc09b315-54d3-44f0-b6ea-8028bd3b212b","Type":"ContainerStarted","Data":"42820d25b8e208745bfe099d0feb7455da75db0c9f4af218313fdf64a2c112a0"}
Nov 03 07:53:18.057443 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:18.057387    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-cnskv" event={"ID":"96cfe445-92c7-4bfe-bafd-8943ce48cee8","Type":"ContainerStarted","Data":"64038c1347be9ac5723deaad2687bfab613f4b65358afc5e62f1bab4c4c1ef4a"}
Nov 03 07:53:18.057549 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:18.057438    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-cnskv" event={"ID":"96cfe445-92c7-4bfe-bafd-8943ce48cee8","Type":"ContainerStarted","Data":"33f8ce7f9bdc0222440db4a7bb237998b388778284b0ce351815314c1edcfd93"}
Nov 03 07:53:18.059077 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:18.059056    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/1.log"
Nov 03 07:53:18.059518 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:18.059493    2579 scope.go:122] "RemoveContainer" containerID="b17a031689eca024fee8f71239c507cd7dc10ca5e0147ff1e9b4d8440b1d8575"
Nov 03 07:53:18.059726 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:18.059675    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5qpnb_openshift-console-operator(d83648de-81dd-4160-8fa9-06d36b38d9de)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" podUID="d83648de-81dd-4160-8fa9-06d36b38d9de"
Nov 03 07:53:18.079799 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:18.077366    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-cnskv" podStartSLOduration=2.077351067 podStartE2EDuration="2.077351067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:18.075929287 +0000 UTC m=+37.943967010" watchObservedRunningTime="2026-11-03 07:53:18.077351067 +0000 UTC m=+37.945388786"
Nov 03 07:53:18.081197 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:18.081173    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fhz8r"]
Nov 03 07:53:18.094091 ip-10-0-140-123 kubenswrapper[2579]: W1103 07:53:18.093865    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1778099a_ebc0_47ba_98d0_2b0e5726e7d2.slice/crio-2ac0604c746ebfcd6adf53fbb1d3da274c8da5401afd683fc4d5d2400398ff87 WatchSource:0}: Error finding container 2ac0604c746ebfcd6adf53fbb1d3da274c8da5401afd683fc4d5d2400398ff87: Status 404 returned error can't find the container with id 2ac0604c746ebfcd6adf53fbb1d3da274c8da5401afd683fc4d5d2400398ff87
Nov 03 07:53:19.066908 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:19.066870    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fhz8r" event={"ID":"1778099a-ebc0-47ba-98d0-2b0e5726e7d2","Type":"ContainerStarted","Data":"2ac0604c746ebfcd6adf53fbb1d3da274c8da5401afd683fc4d5d2400398ff87"}
Nov 03 07:53:19.261849 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:19.261520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:19.261849 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:19.261591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:19.261849 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:19.261636    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 07:53:19.261849 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:19.261714    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-rbac-proxy-cm podName:6731c200-9eda-44f2-aa13-2b4a8fbda85e nodeName:}" failed. No retries permitted until 2026-11-03 07:53:23.261692269 +0000 UTC m=+43.129729981 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wd497" (UID: "6731c200-9eda-44f2-aa13-2b4a8fbda85e") : configmap "kube-rbac-proxy" not found
Nov 03 07:53:19.261849 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:19.261745    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 07:53:19.261849 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:19.261816    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls podName:6731c200-9eda-44f2-aa13-2b4a8fbda85e nodeName:}" failed. No retries permitted until 2026-11-03 07:53:23.261799749 +0000 UTC m=+43.129837472 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wd497" (UID: "6731c200-9eda-44f2-aa13-2b4a8fbda85e") : secret "insights-runtime-extractor-tls" not found
Nov 03 07:53:20.073191 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:20.073152    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-d7zfg" event={"ID":"fc09b315-54d3-44f0-b6ea-8028bd3b212b","Type":"ContainerStarted","Data":"59071b79a6d3879cad138a5d6160a7f8e22010cb85f40fe9d6f32ea3b156d7f3"}
Nov 03 07:53:20.075585 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:20.075546    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fhz8r" event={"ID":"1778099a-ebc0-47ba-98d0-2b0e5726e7d2","Type":"ContainerStarted","Data":"3d79aa29b6f80c2f400198733d5d6fc3756629ec2174ccd96312c9eeae293d89"}
Nov 03 07:53:20.105462 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:20.105420    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-d7zfg" podStartSLOduration=1.240306116 podStartE2EDuration="3.105407296s" totalImagesPullingTime="1.86510118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:20.089259165 +0000 UTC m=+39.957296886" watchObservedRunningTime="2026-11-03 07:53:20.105407296 +0000 UTC m=+39.973445070"
Nov 03 07:53:20.105630 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:20.105611    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fhz8r" podStartSLOduration=1.282645337 podStartE2EDuration="3.105608092s" totalImagesPullingTime="1.822962755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:20.103730754 +0000 UTC m=+39.971768474" watchObservedRunningTime="2026-11-03 07:53:20.105608092 +0000 UTC m=+39.973645812"
Nov 03 07:53:20.320638 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:20.320568    2579 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:20.320638 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:20.320630    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:20.321039 ip-10-0-140-123 kubenswrapper[2579]: I1103 07:53:20.321021    2579 scope.go:122] "RemoveContainer" containerID="b17a031689eca024fee8f71239c507cd7dc10ca5e0147ff1e9b4d8440b1d8575"
Nov 03 07:53:20.321259 ip-10-0-140-123 kubenswrapper[2579]: E1103 07:53:20.321224    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5qpnb_openshift-console-operator(d83648de-81dd-4160-8fa9-06d36b38d9de)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" podUID="d83648de-81dd-4160-8fa9-06d36b38d9de"
Nov 03 07:53:21.837149 ip-10-0-140-123 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 07:53:21.844350 ip-10-0-140-123 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 07:53:21.844517 ip-10-0-140-123 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 07:53:21.844567 ip-10-0-140-123 systemd[1]: kubelet.service: Consumed 3.003s CPU time.
Nov 03 07:53:21.855420 ip-10-0-140-123 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 07:53:21.935139 ip-10-0-140-123 kubenswrapper[5070]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 07:53:21.935139 ip-10-0-140-123 kubenswrapper[5070]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 07:53:21.935139 ip-10-0-140-123 kubenswrapper[5070]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 07:53:21.935522 ip-10-0-140-123 kubenswrapper[5070]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 07:53:21.937228 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937151    5070 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:53:21.937228 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937228    5070 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:53:21.937296 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937232    5070 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:53:21.937296 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937235    5070 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:53:21.937296 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937238    5070 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:53:21.937296 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937241    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:53:21.937296 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937244    5070 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:53:21.937296 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937246    5070 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:53:21.937296 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937249    5070 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:53:21.937296 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937252    5070 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:53:21.937296 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937255    5070 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:53:21.937296 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937258    5070 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:53:21.937296 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937260    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:53:21.937296 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937263    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:53:21.937296 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937266    5070 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:53:21.937296 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937268    5070 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:53:21.937296 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937271    5070 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:53:21.937296 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937273    5070 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:53:21.937296 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937278    5070 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:53:21.937296 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937281    5070 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:53:21.937296 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937284    5070 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:53:21.937782 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937296    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:53:21.937782 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937299    5070 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:53:21.937782 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937302    5070 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:53:21.937782 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937305    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:53:21.937782 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937308    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:53:21.937782 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937311    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:53:21.937782 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937314    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:53:21.937782 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937316    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:53:21.937782 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937319    5070 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:53:21.937782 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937321    5070 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:53:21.937782 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937324    5070 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:53:21.937782 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937327    5070 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:53:21.937782 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937329    5070 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:53:21.937782 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937332    5070 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:53:21.937782 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937334    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:53:21.937782 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937337    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:53:21.937782 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937340    5070 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:53:21.937782 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937342    5070 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:53:21.937782 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937351    5070 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:53:21.937782 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937354    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:53:21.938273 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937356    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:53:21.938273 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937359    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:53:21.938273 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937362    5070 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:53:21.938273 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937365    5070 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:53:21.938273 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937368    5070 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:53:21.938273 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937371    5070 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:53:21.938273 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937374    5070 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:53:21.938273 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937377    5070 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:53:21.938273 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937380    5070 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:53:21.938273 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937382    5070 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:53:21.938273 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937385    5070 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:53:21.938273 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937388    5070 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:53:21.938273 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937390    5070 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:53:21.938273 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937393    5070 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:53:21.938273 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937395    5070 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:53:21.938273 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937398    5070 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:53:21.938273 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937401    5070 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:53:21.938273 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937403    5070 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:53:21.938273 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937406    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:53:21.938731 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937408    5070 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:53:21.938731 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937411    5070 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:53:21.938731 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937413    5070 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:53:21.938731 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937416    5070 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:53:21.938731 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937418    5070 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:53:21.938731 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937421    5070 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:53:21.938731 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937423    5070 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:53:21.938731 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937426    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:53:21.938731 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937428    5070 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:53:21.938731 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937431    5070 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:53:21.938731 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937434    5070 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:53:21.938731 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937436    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:53:21.938731 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937445    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:53:21.938731 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937448    5070 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:53:21.938731 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937451    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:53:21.938731 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937453    5070 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:53:21.938731 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937456    5070 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:53:21.938731 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937459    5070 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:53:21.938731 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937463    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:53:21.938731 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937466    5070 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:53:21.939251 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937468    5070 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:53:21.939251 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937471    5070 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:53:21.939251 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937473    5070 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:53:21.939251 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937476    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:53:21.939251 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937478    5070 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:53:21.939251 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937481    5070 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:53:21.939251 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937484    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:53:21.939251 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937486    5070 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:53:21.939251 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937489    5070 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:53:21.939251 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937491    5070 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:53:21.939251 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937494    5070 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:53:21.939251 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937496    5070 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:53:21.939251 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937499    5070 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:53:21.939251 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937502    5070 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:53:21.939251 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937504    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:53:21.939251 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937507    5070 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:53:21.939251 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937509    5070 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:53:21.939251 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937512    5070 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:53:21.939251 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937515    5070 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:53:21.939251 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937518    5070 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:53:21.939747 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937520    5070 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:53:21.939747 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937523    5070 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:53:21.939747 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937526    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:53:21.939747 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937531    5070 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:53:21.939747 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937534    5070 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:53:21.939747 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.937559    5070 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:53:21.939747 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938352    5070 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:53:21.939747 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938361    5070 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:53:21.939747 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938365    5070 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:53:21.939747 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938371    5070 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:53:21.939747 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938374    5070 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:53:21.939747 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938378    5070 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:53:21.939747 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938381    5070 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:53:21.939747 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938383    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:53:21.939747 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938386    5070 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:53:21.939747 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938389    5070 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:53:21.939747 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938391    5070 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:53:21.939747 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938394    5070 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:53:21.939747 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938396    5070 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:53:21.940253 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938399    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:53:21.940253 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938402    5070 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:53:21.940253 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938405    5070 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:53:21.940253 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938407    5070 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:53:21.940253 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938410    5070 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:53:21.940253 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938413    5070 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:53:21.940253 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938415    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:53:21.940253 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938418    5070 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:53:21.940253 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938420    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:53:21.940253 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938423    5070 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:53:21.940253 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938426    5070 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:53:21.940253 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938428    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:53:21.940253 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938431    5070 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:53:21.940253 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938434    5070 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:53:21.940253 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938436    5070 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:53:21.940253 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938439    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:53:21.940253 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938442    5070 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:53:21.940253 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938444    5070 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:53:21.940253 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938447    5070 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:53:21.940253 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938450    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:53:21.940745 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938452    5070 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:53:21.940745 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938455    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:53:21.940745 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938458    5070 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:53:21.940745 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938460    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:53:21.940745 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938463    5070 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:53:21.940745 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938466    5070 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:53:21.940745 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938469    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:53:21.940745 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938471    5070 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:53:21.940745 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938474    5070 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:53:21.940745 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938476    5070 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:53:21.940745 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938479    5070 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:53:21.940745 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938481    5070 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:53:21.940745 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938484    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:53:21.940745 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938486    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:53:21.940745 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938489    5070 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:53:21.940745 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938491    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:53:21.940745 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938494    5070 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:53:21.940745 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938496    5070 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:53:21.940745 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938499    5070 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:53:21.941271 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938503    5070 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:53:21.941271 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938508    5070 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:53:21.941271 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938511    5070 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:53:21.941271 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938514    5070 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:53:21.941271 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938517    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:53:21.941271 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938520    5070 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:53:21.941271 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938523    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:53:21.941271 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938525    5070 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:53:21.941271 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938528    5070 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:53:21.941271 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938531    5070 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:53:21.941271 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938533    5070 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:53:21.941271 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938536    5070 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:53:21.941271 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938538    5070 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:53:21.941271 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938542    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:53:21.941271 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938544    5070 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:53:21.941271 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938548    5070 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:53:21.941271 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938550    5070 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:53:21.941271 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938553    5070 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:53:21.941271 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938555    5070 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:53:21.941732 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938558    5070 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:53:21.941732 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938561    5070 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:53:21.941732 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938563    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:53:21.941732 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938566    5070 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:53:21.941732 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938568    5070 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:53:21.941732 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938571    5070 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:53:21.941732 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938574    5070 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:53:21.941732 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938576    5070 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:53:21.941732 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938579    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:53:21.941732 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938582    5070 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:53:21.941732 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938585    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:53:21.941732 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938587    5070 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:53:21.941732 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938589    5070 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:53:21.941732 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938592    5070 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:53:21.941732 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938595    5070 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:53:21.941732 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938597    5070 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:53:21.941732 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938600    5070 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:53:21.941732 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938602    5070 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:53:21.941732 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938605    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:53:21.941732 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938607    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:53:21.942272 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938609    5070 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:53:21.942272 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938612    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:53:21.942272 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938614    5070 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:53:21.942272 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938617    5070 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:53:21.942272 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938619    5070 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:53:21.942272 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938622    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:53:21.942272 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938624    5070 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:53:21.942272 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938631    5070 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:53:21.942272 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938634    5070 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:53:21.942272 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938637    5070 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:53:21.942272 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938639    5070 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:53:21.942272 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938642    5070 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:53:21.942272 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938645    5070 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:53:21.942272 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938648    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:53:21.942272 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.938651    5070 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:53:21.942272 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939144    5070 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:53:21.942272 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939150    5070 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:53:21.942272 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939153    5070 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:53:21.942272 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939156    5070 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:53:21.942272 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939159    5070 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:53:21.942838 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939162    5070 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:53:21.942838 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939165    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:53:21.942838 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939168    5070 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:53:21.942838 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939171    5070 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:53:21.942838 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939173    5070 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:53:21.942838 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939176    5070 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:53:21.942838 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939179    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:53:21.942838 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939181    5070 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:53:21.942838 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939184    5070 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:53:21.942838 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939186    5070 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:53:21.942838 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939189    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:53:21.942838 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939191    5070 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:53:21.942838 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939194    5070 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:53:21.942838 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939196    5070 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:53:21.942838 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939199    5070 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:53:21.942838 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939201    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:53:21.942838 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939204    5070 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:53:21.942838 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939206    5070 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:53:21.942838 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939209    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:53:21.943685 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939214    5070 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:53:21.943685 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939218    5070 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:53:21.943685 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939220    5070 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:53:21.943685 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939223    5070 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:53:21.943685 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939225    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:53:21.943685 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939228    5070 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:53:21.943685 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939230    5070 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:53:21.943685 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939233    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:53:21.943685 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939235    5070 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:53:21.943685 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939238    5070 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:53:21.943685 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939240    5070 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:53:21.943685 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939243    5070 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:53:21.943685 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939245    5070 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:53:21.943685 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939248    5070 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:53:21.943685 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939251    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:53:21.943685 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939253    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:53:21.943685 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939263    5070 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:53:21.943685 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939266    5070 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:53:21.943685 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939269    5070 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:53:21.944526 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939273    5070 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:53:21.944526 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939275    5070 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:53:21.944526 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939278    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:53:21.944526 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939281    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:53:21.944526 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939283    5070 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:53:21.944526 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939286    5070 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:53:21.944526 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939288    5070 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:53:21.944526 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939291    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:53:21.944526 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939293    5070 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:53:21.944526 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939296    5070 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:53:21.944526 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939299    5070 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:53:21.944526 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939302    5070 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:53:21.944526 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939304    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:53:21.944526 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939308    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:53:21.944526 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939316    5070 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:53:21.944526 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939319    5070 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:53:21.944526 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939322    5070 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:53:21.944526 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939326    5070 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:53:21.944526 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939330    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:53:21.945353 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939332    5070 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:53:21.945353 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939335    5070 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:53:21.945353 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939338    5070 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:53:21.945353 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939340    5070 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:53:21.945353 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939343    5070 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:53:21.945353 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939345    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:53:21.945353 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939348    5070 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:53:21.945353 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939350    5070 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:53:21.945353 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939353    5070 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:53:21.945353 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939355    5070 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:53:21.945353 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939358    5070 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:53:21.945353 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939360    5070 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:53:21.945353 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939364    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:53:21.945353 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939366    5070 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:53:21.945353 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939369    5070 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:53:21.945353 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939371    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:53:21.945353 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939373    5070 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:53:21.945353 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939376    5070 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:53:21.945353 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939379    5070 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:53:21.945353 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939381    5070 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:53:21.946252 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939384    5070 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:53:21.946252 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939386    5070 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:53:21.946252 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939389    5070 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:53:21.946252 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939391    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:53:21.946252 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939394    5070 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:53:21.946252 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939396    5070 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:53:21.946252 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939400    5070 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:53:21.946252 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939405    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:53:21.946252 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939408    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:53:21.946252 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939411    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:53:21.946252 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939413    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:53:21.946252 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939416    5070 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:53:21.946252 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939420    5070 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:53:21.946252 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939424    5070 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:53:21.946252 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939427    5070 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:53:21.946252 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939430    5070 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:53:21.946252 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939432    5070 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:53:21.946252 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939435    5070 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:53:21.946252 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939438    5070 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939441    5070 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939443    5070 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939446    5070 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939449    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.939452    5070 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939548    5070 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939558    5070 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939568    5070 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939575    5070 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939580    5070 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939583    5070 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939588    5070 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939592    5070 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939595    5070 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939598    5070 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939602    5070 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939605    5070 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939608    5070 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939611    5070 flags.go:64] FLAG: --cgroup-root=""
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939617    5070 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939624    5070 flags.go:64] FLAG: --client-ca-file=""
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939627    5070 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939632    5070 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 07:53:21.947113 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939636    5070 flags.go:64] FLAG: --cluster-domain=""
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939639    5070 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939642    5070 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939646    5070 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939649    5070 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939653    5070 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939656    5070 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939659    5070 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939662    5070 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939665    5070 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939668    5070 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939671    5070 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939674    5070 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939677    5070 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939685    5070 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939688    5070 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939691    5070 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939694    5070 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939697    5070 flags.go:64] FLAG: --enable-server="true"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939700    5070 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939704    5070 flags.go:64] FLAG: --event-burst="100"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939708    5070 flags.go:64] FLAG: --event-qps="50"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939710    5070 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939714    5070 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 07:53:21.947955 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939717    5070 flags.go:64] FLAG: --eviction-hard=""
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939721    5070 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939724    5070 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939727    5070 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939732    5070 flags.go:64] FLAG: --eviction-soft=""
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939736    5070 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939739    5070 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939742    5070 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939745    5070 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939748    5070 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939751    5070 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939766    5070 flags.go:64] FLAG: --feature-gates=""
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939771    5070 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939774    5070 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939778    5070 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939781    5070 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939784    5070 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939787    5070 flags.go:64] FLAG: --help="false"
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939790    5070 flags.go:64] FLAG: --hostname-override="ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939793    5070 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939796    5070 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939799    5070 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939802    5070 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 07:53:21.948549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939806    5070 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939809    5070 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939811    5070 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939814    5070 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939817    5070 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939820    5070 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939824    5070 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939827    5070 flags.go:64] FLAG: --kube-reserved=""
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939830    5070 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939833    5070 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939836    5070 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939839    5070 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939842    5070 flags.go:64] FLAG: --lock-file=""
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939845    5070 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939849    5070 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939852    5070 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939858    5070 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939860    5070 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939864    5070 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939867    5070 flags.go:64] FLAG: --logging-format="text"
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939870    5070 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939873    5070 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939876    5070 flags.go:64] FLAG: --manifest-url=""
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939879    5070 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939884    5070 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 07:53:21.949156 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939887    5070 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939890    5070 flags.go:64] FLAG: --max-pods="110"
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939893    5070 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939896    5070 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939899    5070 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939902    5070 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939905    5070 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939908    5070 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939911    5070 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939918    5070 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939921    5070 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939924    5070 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939927    5070 flags.go:64] FLAG: --pod-cidr=""
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939930    5070 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939933    5070 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939936    5070 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939939    5070 flags.go:64] FLAG: --port="10250"
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939942    5070 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939945    5070 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0c05fecb68435fa38"
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939948    5070 flags.go:64] FLAG: --qos-reserved=""
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939951    5070 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939955    5070 flags.go:64] FLAG: --register-node="true"
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939958    5070 flags.go:64] FLAG: --register-with-taints=""
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939962    5070 flags.go:64] FLAG: --registry-burst="10"
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939965    5070 flags.go:64] FLAG: --registry-qps="5"
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939968    5070 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 07:53:21.949887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939971    5070 flags.go:64] FLAG: --reserved-memory=""
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939975    5070 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939978    5070 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939981    5070 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939984    5070 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939987    5070 flags.go:64] FLAG: --runonce="false"
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939990    5070 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939993    5070 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939996    5070 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.939999    5070 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940002    5070 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940005    5070 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940008    5070 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940011    5070 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940014    5070 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940017    5070 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940020    5070 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940022    5070 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940025    5070 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940028    5070 flags.go:64] FLAG: --system-cgroups=""
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940031    5070 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940036    5070 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940039    5070 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940042    5070 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940046    5070 flags.go:64] FLAG: --tls-min-version=""
Nov 03 07:53:21.950520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940049    5070 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 07:53:21.951248 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940052    5070 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 07:53:21.951248 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940056    5070 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 07:53:21.951248 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940059    5070 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 07:53:21.951248 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940062    5070 flags.go:64] FLAG: --v="2"
Nov 03 07:53:21.951248 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940067    5070 flags.go:64] FLAG: --version="false"
Nov 03 07:53:21.951248 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940071    5070 flags.go:64] FLAG: --vmodule=""
Nov 03 07:53:21.951248 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940075    5070 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 07:53:21.951248 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940079    5070 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 07:53:21.951248 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940210    5070 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:53:21.951248 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940213    5070 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:53:21.951248 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940216    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:53:21.951248 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940219    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:53:21.951248 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940221    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:53:21.951248 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940224    5070 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:53:21.951248 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940227    5070 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:53:21.951248 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940229    5070 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:53:21.951248 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940232    5070 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:53:21.951248 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940235    5070 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:53:21.951248 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940237    5070 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:53:21.951248 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940240    5070 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:53:21.951248 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940243    5070 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:53:21.951883 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940247    5070 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:53:21.951883 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940250    5070 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:53:21.951883 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940253    5070 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:53:21.951883 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940256    5070 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:53:21.951883 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940259    5070 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:53:21.951883 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940261    5070 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:53:21.951883 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940264    5070 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:53:21.951883 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940268    5070 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:53:21.951883 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940272    5070 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:53:21.951883 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940275    5070 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:53:21.951883 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940279    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:53:21.951883 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940282    5070 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:53:21.951883 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940284    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:53:21.951883 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940287    5070 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:53:21.951883 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940290    5070 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:53:21.951883 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940292    5070 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:53:21.951883 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940295    5070 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:53:21.951883 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940297    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:53:21.951883 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940300    5070 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:53:21.952585 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940303    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:53:21.952585 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940306    5070 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:53:21.952585 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940308    5070 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:53:21.952585 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940311    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:53:21.952585 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940313    5070 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:53:21.952585 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940316    5070 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:53:21.952585 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940318    5070 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:53:21.952585 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940321    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:53:21.952585 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940323    5070 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:53:21.952585 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940326    5070 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:53:21.952585 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940328    5070 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:53:21.952585 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940331    5070 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:53:21.952585 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940333    5070 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:53:21.952585 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940336    5070 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:53:21.952585 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940338    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:53:21.952585 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940341    5070 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:53:21.952585 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940345    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:53:21.952585 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940347    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:53:21.952585 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940350    5070 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:53:21.953443 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940352    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:53:21.953443 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940355    5070 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:53:21.953443 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940358    5070 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:53:21.953443 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940360    5070 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:53:21.953443 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940364    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:53:21.953443 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940367    5070 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:53:21.953443 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940370    5070 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:53:21.953443 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940373    5070 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:53:21.953443 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940375    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:53:21.953443 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940378    5070 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:53:21.953443 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940381    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:53:21.953443 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940383    5070 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:53:21.953443 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940386    5070 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:53:21.953443 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940388    5070 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:53:21.953443 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940391    5070 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:53:21.953443 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940394    5070 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:53:21.953443 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940397    5070 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:53:21.953443 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940399    5070 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:53:21.953443 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940402    5070 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:53:21.953443 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940405    5070 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:53:21.954282 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940407    5070 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:53:21.954282 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940410    5070 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:53:21.954282 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940412    5070 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:53:21.954282 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940415    5070 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:53:21.954282 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940417    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:53:21.954282 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940420    5070 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:53:21.954282 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940422    5070 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:53:21.954282 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940425    5070 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:53:21.954282 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940427    5070 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:53:21.954282 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940431    5070 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:53:21.954282 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940433    5070 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:53:21.954282 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940436    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:53:21.954282 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940438    5070 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:53:21.954282 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940441    5070 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:53:21.954282 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940443    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:53:21.954282 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940446    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:53:21.954282 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940450    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:53:21.954282 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940452    5070 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:53:21.954282 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940455    5070 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:53:21.954282 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940458    5070 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:53:21.955189 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940460    5070 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:53:21.955189 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940463    5070 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:53:21.955189 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940465    5070 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:53:21.955189 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940468    5070 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:53:21.955189 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940470    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:53:21.955189 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940473    5070 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:53:21.955189 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940475    5070 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:53:21.955189 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940478    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:53:21.955189 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940481    5070 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:53:21.955189 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940483    5070 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:53:21.955189 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940486    5070 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:53:21.955189 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940488    5070 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:53:21.955189 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940491    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:53:21.955189 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940493    5070 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:53:21.955189 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940496    5070 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:53:21.955189 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.940521    5070 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 07:53:21.955701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940556    5070 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:53:21.955701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940560    5070 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:53:21.955701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940563    5070 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:53:21.955701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940567    5070 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:53:21.955701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940569    5070 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:53:21.955701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940574    5070 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:53:21.955701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940577    5070 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:53:21.955701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940582    5070 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:53:21.955701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940585    5070 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:53:21.955701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940588    5070 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:53:21.955701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940591    5070 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:53:21.955701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940593    5070 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:53:21.955701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940596    5070 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:53:21.955701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940601    5070 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:53:21.955701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940605    5070 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:53:21.955701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940608    5070 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:53:21.955701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940611    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:53:21.955701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940613    5070 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:53:21.955701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940616    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:53:21.956419 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940618    5070 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:53:21.956419 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940621    5070 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:53:21.956419 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940623    5070 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:53:21.956419 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940626    5070 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:53:21.956419 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940628    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:53:21.956419 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940631    5070 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:53:21.956419 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940634    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:53:21.956419 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940636    5070 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:53:21.956419 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940639    5070 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:53:21.956419 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940641    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:53:21.956419 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940644    5070 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:53:21.956419 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940646    5070 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:53:21.956419 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940649    5070 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:53:21.956419 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940651    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:53:21.956419 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940654    5070 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:53:21.956419 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940656    5070 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:53:21.956419 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940659    5070 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:53:21.956419 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940661    5070 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:53:21.956419 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940665    5070 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:53:21.956923 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940668    5070 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:53:21.956923 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940670    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:53:21.956923 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940673    5070 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:53:21.956923 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940676    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:53:21.956923 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940678    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:53:21.956923 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940681    5070 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:53:21.956923 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940683    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:53:21.956923 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940687    5070 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:53:21.956923 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940690    5070 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:53:21.956923 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940692    5070 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:53:21.956923 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940695    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:53:21.956923 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940698    5070 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:53:21.956923 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940701    5070 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:53:21.956923 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940703    5070 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:53:21.956923 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940706    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:53:21.956923 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940709    5070 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:53:21.956923 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940711    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:53:21.956923 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940714    5070 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:53:21.956923 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940716    5070 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:53:21.956923 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940719    5070 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:53:21.957466 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940722    5070 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:53:21.957466 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940725    5070 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:53:21.957466 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940728    5070 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:53:21.957466 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940730    5070 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:53:21.957466 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940733    5070 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:53:21.957466 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940735    5070 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:53:21.957466 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940738    5070 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:53:21.957466 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940740    5070 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:53:21.957466 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940743    5070 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:53:21.957466 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940745    5070 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:53:21.957466 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940748    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:53:21.957466 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940752    5070 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:53:21.957466 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940766    5070 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:53:21.957466 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940769    5070 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:53:21.957466 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940772    5070 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:53:21.957466 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940775    5070 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:53:21.957466 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940777    5070 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:53:21.957466 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940780    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:53:21.957466 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940782    5070 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:53:21.957466 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940787    5070 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:53:21.957971 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940789    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:53:21.957971 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940792    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:53:21.957971 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940795    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:53:21.957971 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940797    5070 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:53:21.957971 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940800    5070 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:53:21.957971 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940803    5070 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:53:21.957971 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940805    5070 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:53:21.957971 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940808    5070 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:53:21.957971 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940810    5070 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:53:21.957971 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940813    5070 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:53:21.957971 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940815    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:53:21.957971 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940818    5070 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:53:21.957971 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940824    5070 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:53:21.957971 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940826    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:53:21.957971 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940829    5070 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:53:21.957971 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940832    5070 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:53:21.957971 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940834    5070 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:53:21.957971 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940837    5070 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:53:21.957971 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940853    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:53:21.958427 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940857    5070 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:53:21.958427 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940860    5070 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:53:21.958427 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940863    5070 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:53:21.958427 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940865    5070 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:53:21.958427 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940869    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:53:21.958427 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940872    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:53:21.958427 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940875    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:53:21.958427 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940877    5070 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:53:21.958427 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.940881    5070 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:53:21.958427 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.945233    5070 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 07:53:21.958427 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.945253    5070 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 07:53:21.958427 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.945276    5070 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 07:53:21.958427 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.945284    5070 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 07:53:21.958427 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945365    5070 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:53:21.958427 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945372    5070 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:53:21.958427 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945377    5070 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:53:21.958427 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945381    5070 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:53:21.958427 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945386    5070 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:53:21.958427 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945391    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:53:21.959037 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945396    5070 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:53:21.959037 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945400    5070 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:53:21.959037 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945404    5070 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:53:21.959037 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945409    5070 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:53:21.959037 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945413    5070 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:53:21.959037 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945417    5070 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:53:21.959037 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945421    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:53:21.959037 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945425    5070 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:53:21.959037 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945430    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:53:21.959037 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945435    5070 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:53:21.959037 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945439    5070 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:53:21.959037 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945444    5070 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:53:21.959037 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945448    5070 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:53:21.959037 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945453    5070 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:53:21.959037 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945457    5070 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:53:21.959037 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945462    5070 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:53:21.959037 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945466    5070 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:53:21.959037 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945471    5070 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:53:21.959037 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945476    5070 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:53:21.959037 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945480    5070 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:53:21.959659 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945484    5070 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:53:21.959659 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945489    5070 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:53:21.959659 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945493    5070 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:53:21.959659 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945498    5070 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:53:21.959659 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945502    5070 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:53:21.959659 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945507    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:53:21.959659 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945512    5070 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:53:21.959659 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945516    5070 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:53:21.959659 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945521    5070 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:53:21.959659 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945525    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:53:21.959659 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945529    5070 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:53:21.959659 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945534    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:53:21.959659 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945538    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:53:21.959659 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945542    5070 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:53:21.959659 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945546    5070 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:53:21.959659 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945551    5070 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:53:21.959659 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945558    5070 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:53:21.959659 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945564    5070 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:53:21.959659 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945569    5070 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:53:21.960193 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945573    5070 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:53:21.960193 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945578    5070 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:53:21.960193 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945583    5070 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:53:21.960193 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945587    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:53:21.960193 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945592    5070 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:53:21.960193 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945596    5070 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:53:21.960193 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945600    5070 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:53:21.960193 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945604    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:53:21.960193 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945609    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:53:21.960193 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945613    5070 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:53:21.960193 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945617    5070 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:53:21.960193 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945621    5070 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:53:21.960193 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945626    5070 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:53:21.960193 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945630    5070 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:53:21.960193 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945634    5070 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:53:21.960193 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945639    5070 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:53:21.960193 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945643    5070 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:53:21.960193 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945648    5070 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:53:21.960193 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945653    5070 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:53:21.960660 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945657    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:53:21.960660 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945661    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:53:21.960660 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945665    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:53:21.960660 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945670    5070 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:53:21.960660 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945674    5070 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:53:21.960660 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945678    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:53:21.960660 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945682    5070 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:53:21.960660 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945686    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:53:21.960660 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945691    5070 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:53:21.960660 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945695    5070 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:53:21.960660 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945699    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:53:21.960660 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945703    5070 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:53:21.960660 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945708    5070 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:53:21.960660 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945712    5070 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:53:21.960660 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945716    5070 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:53:21.960660 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945721    5070 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:53:21.960660 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945725    5070 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:53:21.960660 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945729    5070 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:53:21.960660 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945734    5070 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:53:21.961228 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945741    5070 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:53:21.961228 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945747    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:53:21.961228 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945752    5070 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:53:21.961228 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945781    5070 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:53:21.961228 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945786    5070 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:53:21.961228 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945791    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:53:21.961228 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945794    5070 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:53:21.961228 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945799    5070 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:53:21.961228 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945803    5070 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:53:21.961228 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945808    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:53:21.961228 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945812    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:53:21.961228 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945817    5070 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:53:21.961228 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945823    5070 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:53:21.961228 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945827    5070 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:53:21.961228 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945831    5070 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:53:21.961228 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945835    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:53:21.961228 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945840    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:53:21.961228 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945844    5070 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:53:21.961228 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945849    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:53:21.961797 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945853    5070 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:53:21.961797 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945857    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:53:21.961797 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945861    5070 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:53:21.961797 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.945867    5070 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:53:21.961797 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.945910    5070 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 07:53:21.961797 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946343    5070 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:53:21.961797 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946357    5070 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:53:21.961797 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946363    5070 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:53:21.961797 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946367    5070 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:53:21.961797 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946373    5070 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:53:21.961797 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946377    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:53:21.961797 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946382    5070 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:53:21.961797 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946393    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:53:21.961797 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946398    5070 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:53:21.961797 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946402    5070 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:53:21.961797 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946407    5070 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:53:21.962171 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946412    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:53:21.962171 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946417    5070 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:53:21.962171 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946421    5070 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:53:21.962171 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946431    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:53:21.962171 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946435    5070 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:53:21.962171 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946440    5070 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:53:21.962171 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946445    5070 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:53:21.962171 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946450    5070 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:53:21.962171 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946454    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:53:21.962171 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946464    5070 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:53:21.962171 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946477    5070 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:53:21.962171 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946482    5070 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:53:21.962171 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946487    5070 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:53:21.962171 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946491    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:53:21.962171 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946496    5070 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:53:21.962171 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946500    5070 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:53:21.962171 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946505    5070 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:53:21.962171 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946509    5070 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:53:21.962171 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946514    5070 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:53:21.962171 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946518    5070 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:53:21.962645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946523    5070 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:53:21.962645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946527    5070 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:53:21.962645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946537    5070 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:53:21.962645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946541    5070 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:53:21.962645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946546    5070 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:53:21.962645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946551    5070 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:53:21.962645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946557    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:53:21.962645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946561    5070 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:53:21.962645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946565    5070 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:53:21.962645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946570    5070 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:53:21.962645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946575    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:53:21.962645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946579    5070 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:53:21.962645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946584    5070 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:53:21.962645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946589    5070 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:53:21.962645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946598    5070 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:53:21.962645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946605    5070 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:53:21.962645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946610    5070 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:53:21.962645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946615    5070 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:53:21.962645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946620    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:53:21.962645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946624    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:53:21.963210 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946629    5070 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:53:21.963210 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946633    5070 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:53:21.963210 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946638    5070 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:53:21.963210 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946643    5070 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:53:21.963210 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946648    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:53:21.963210 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946652    5070 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:53:21.963210 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946662    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:53:21.963210 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946667    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:53:21.963210 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946671    5070 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:53:21.963210 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946676    5070 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:53:21.963210 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946680    5070 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:53:21.963210 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946685    5070 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:53:21.963210 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946689    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:53:21.963210 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946694    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:53:21.963210 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946699    5070 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:53:21.963210 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946703    5070 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:53:21.963210 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946708    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:53:21.963210 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946713    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:53:21.963210 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946717    5070 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:53:21.963701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946726    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:53:21.963701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946731    5070 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:53:21.963701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946736    5070 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:53:21.963701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946741    5070 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:53:21.963701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946746    5070 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:53:21.963701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946750    5070 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:53:21.963701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946771    5070 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:53:21.963701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946776    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:53:21.963701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946783    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:53:21.963701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946787    5070 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:53:21.963701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946792    5070 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:53:21.963701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946796    5070 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:53:21.963701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946801    5070 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:53:21.963701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946810    5070 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:53:21.963701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946815    5070 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:53:21.963701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946820    5070 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:53:21.963701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.946824    5070 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:53:21.963701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947321    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:53:21.963701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947473    5070 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:53:21.963701 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947479    5070 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:53:21.964249 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947482    5070 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:53:21.964249 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947487    5070 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:53:21.964249 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947492    5070 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:53:21.964249 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947496    5070 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:53:21.964249 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947499    5070 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:53:21.964249 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947502    5070 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:53:21.964249 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947504    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:53:21.964249 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947507    5070 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:53:21.964249 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947510    5070 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:53:21.964249 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947512    5070 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:53:21.964249 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947516    5070 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:53:21.964249 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947520    5070 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:53:21.964249 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947524    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:53:21.964249 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947527    5070 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:53:21.964249 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947531    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:53:21.964249 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947534    5070 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:53:21.964645 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.947569    5070 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 07:53:21.964645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947608    5070 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 07:53:21.964645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947613    5070 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 07:53:21.964645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947617    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 07:53:21.964645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947619    5070 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 07:53:21.964645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947622    5070 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 07:53:21.964645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947625    5070 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 07:53:21.964645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947629    5070 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 07:53:21.964645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947633    5070 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 07:53:21.964645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947636    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 07:53:21.964645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947639    5070 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 07:53:21.964645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947642    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 07:53:21.964645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947644    5070 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 07:53:21.964645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947647    5070 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 07:53:21.964645 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947651    5070 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 07:53:21.965068 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947653    5070 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 07:53:21.965068 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947656    5070 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 07:53:21.965068 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947658    5070 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 07:53:21.965068 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947661    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 07:53:21.965068 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947664    5070 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 07:53:21.965068 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947666    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 07:53:21.965068 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947669    5070 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 07:53:21.965068 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947672    5070 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 07:53:21.965068 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947675    5070 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 07:53:21.965068 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947678    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 07:53:21.965068 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947680    5070 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 07:53:21.965068 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947683    5070 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 07:53:21.965068 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947686    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 07:53:21.965068 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947688    5070 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 07:53:21.965068 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947691    5070 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 07:53:21.965068 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947697    5070 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 07:53:21.965068 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947700    5070 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 07:53:21.965068 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947703    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 07:53:21.965068 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947705    5070 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 07:53:21.965528 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947708    5070 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 07:53:21.965528 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947710    5070 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 07:53:21.965528 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947713    5070 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 07:53:21.965528 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947716    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 07:53:21.965528 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947718    5070 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 07:53:21.965528 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947721    5070 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 07:53:21.965528 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947723    5070 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 07:53:21.965528 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947726    5070 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 07:53:21.965528 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947729    5070 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 07:53:21.965528 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947731    5070 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 07:53:21.965528 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947734    5070 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 07:53:21.965528 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947736    5070 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 07:53:21.965528 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947739    5070 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 07:53:21.965528 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947742    5070 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 07:53:21.965528 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947745    5070 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 07:53:21.965528 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947747    5070 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 07:53:21.965528 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947750    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 07:53:21.965528 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947753    5070 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 07:53:21.965528 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947771    5070 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 07:53:21.965528 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947775    5070 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 07:53:21.966078 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947778    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 07:53:21.966078 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947781    5070 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 07:53:21.966078 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947784    5070 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 07:53:21.966078 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947788    5070 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 07:53:21.966078 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947791    5070 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 07:53:21.966078 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947794    5070 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 07:53:21.966078 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947797    5070 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 07:53:21.966078 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947799    5070 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 07:53:21.966078 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947806    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 07:53:21.966078 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947809    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 07:53:21.966078 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947812    5070 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 07:53:21.966078 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947815    5070 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 07:53:21.966078 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947818    5070 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 07:53:21.966078 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947821    5070 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 07:53:21.966078 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947823    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 07:53:21.966078 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947826    5070 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 07:53:21.966078 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947828    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 07:53:21.966078 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947831    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 07:53:21.966078 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947834    5070 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 07:53:21.966634 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947836    5070 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 07:53:21.966634 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947839    5070 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 07:53:21.966634 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947842    5070 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 07:53:21.966634 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947845    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 07:53:21.966634 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947848    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 07:53:21.966634 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947851    5070 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 07:53:21.966634 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947853    5070 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 07:53:21.966634 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947856    5070 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 07:53:21.966634 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947858    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 07:53:21.966634 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947861    5070 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 07:53:21.966634 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947864    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 07:53:21.966634 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947866    5070 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 07:53:21.966634 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947869    5070 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 07:53:21.966634 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947872    5070 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 07:53:21.966634 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947875    5070 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 07:53:21.966634 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947877    5070 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 07:53:21.966634 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947880    5070 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 07:53:21.966634 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947882    5070 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 07:53:21.966634 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947885    5070 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 07:53:21.966634 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947887    5070 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 07:53:21.967499 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947890    5070 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 07:53:21.967499 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947895    5070 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 07:53:21.967499 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947897    5070 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 07:53:21.967499 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947900    5070 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 07:53:21.967499 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947903    5070 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 07:53:21.967499 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947905    5070 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 07:53:21.967499 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947907    5070 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 07:53:21.967499 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947910    5070 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 07:53:21.967499 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947913    5070 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 07:53:21.967499 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947915    5070 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 07:53:21.967499 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947918    5070 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 07:53:21.967499 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947923    5070 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 07:53:21.967499 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947927    5070 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 07:53:21.967499 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:21.947930    5070 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 07:53:21.967499 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.947986    5070 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 07:53:21.967499 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.948932    5070 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 07:53:21.967499 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.948984    5070 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 07:53:21.967499 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.949155    5070 server.go:1014] "Starting client certificate rotation"
Nov 03 07:53:21.967499 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.949293    5070 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 07:53:21.968268 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.949345    5070 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 07:47:40 +0000 UTC" deadline="2028-04-04 09:59:26.281300316 +0000 UTC"
Nov 03 07:53:21.968268 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.949370    5070 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12434h6m4.331931649s"
Nov 03 07:53:21.968268 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.949586    5070 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 07:53:21.968268 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.950598    5070 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 07:53:21.968268 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.951644    5070 log.go:25] "Validated CRI v1 runtime API"
Nov 03 07:53:21.968268 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.954973    5070 log.go:25] "Validated CRI v1 image API"
Nov 03 07:53:21.968268 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.955878    5070 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 07:53:21.968268 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.958064    5070 fs.go:135] Filesystem UUIDs: map[37e0c1fc-d3c8-4ff0-9a4d-34d6d14b78c3:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d182e259-d93c-4f5d-bb2f-7384d6159fd3:/dev/nvme0n1p3]
Nov 03 07:53:21.968681 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.958091    5070 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f773c5bcdf6c9660c4dab8c6b494a5750da8e38e2843d8662d052cdc2b59df0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f773c5bcdf6c9660c4dab8c6b494a5750da8e38e2843d8662d052cdc2b59df0/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e702f08ed7ed20f697b4723e95f5b16c80b7b124dab15c88b47143f1709ef83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e702f08ed7ed20f697b4723e95f5b16c80b7b124dab15c88b47143f1709ef83/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ead0f5b21df840f477ce3be72c0d8eb6c637190655753167aa94fc5aa524f3e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ead0f5b21df840f477ce3be72c0d8eb6c637190655753167aa94fc5aa524f3e/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/20c64d2f5beeb6f50908c2f3e422d3393f3829ca8d4366021ef2f482b4794767/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/20c64d2f5beeb6f50908c2f3e422d3393f3829ca8d4366021ef2f482b4794767/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ac0604c746ebfcd6adf53fbb1d3da274c8da5401afd683fc4d5d2400398ff87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ac0604c746ebfcd6adf53fbb1d3da274c8da5401afd683fc4d5d2400398ff87/userdata/shm major:0 minor:366 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2c89b4b61002542a42a86b9447902783481924f92225ac67a1eeccf1f1af5b61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2c89b4b61002542a42a86b9447902783481924f92225ac67a1eeccf1f1af5b61/userdata/shm major:0 minor:227 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/311618f3e5f892bfbf815f887a7944193e5a2914303767dc0afd89d90983ec68/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/311618f3e5f892bfbf815f887a7944193e5a2914303767dc0afd89d90983ec68/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32284d065e9cfd2ff3d1d763bb2440817af436a19a096e3d43956a2a7429f7e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32284d065e9cfd2ff3d1d763bb2440817af436a19a096e3d43956a2a7429f7e8/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33f0b316861516f8b3fea6f55511942d9d0c997927043f4ea9ed2846ec122015/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33f0b316861516f8b3fea6f55511942d9d0c997927043f4ea9ed2846ec122015/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/33f8ce7f9bdc0222440db4a7bb237998b388778284b0ce351815314c1edcfd93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/33f8ce7f9bdc0222440db4a7bb237998b388778284b0ce351815314c1edcfd93/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3db1086282f68b516f21655c635c9ae9a15e019a958eeb27efacbb40a9e3bf53/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3db1086282f68b516f21655c635c9ae9a15e019a958eeb27efacbb40a9e3bf53/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/414453077f111a09b67bb769a7b7b052ca00eeeba4a59290085399c8f9b2ceed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/414453077f111a09b67bb769a7b7b052ca00eeeba4a59290085399c8f9b2ceed/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/424d19eae090d708343b986bdafa6c2807fa7980e6a4e15c660c8513e9dc243b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/424d19eae090d708343b986bdafa6c2807fa7980e6a4e15c660c8513e9dc243b/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42820d25b8e208745bfe099d0feb7455da75db0c9f4af218313fdf64a2c112a0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42820d25b8e208745bfe099d0feb7455da75db0c9f4af218313fdf64a2c112a0/userdata/shm major:0 minor:364 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7378c45cd11e438cb3bb6d12e79041ef787a8b5093548efa0943413db3b2d294/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7378c45cd11e438cb3bb6d12e79041ef787a8b5093548efa0943413db3b2d294/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/910927b265f7362ee19a8aa7142206e182ead12e4de172025dc36d0ff37c70cb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/910927b265f7362ee19a8aa7142206e182ead12e4de172025dc36d0ff37c70cb/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c0e481f1c0afebf976b20aa412a47bc61e96b6a980d3ca65fdde537b1ff35ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c0e481f1c0afebf976b20aa412a47bc61e96b6a980d3ca65fdde537b1ff35ce/userdata/shm major:0 minor:316 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a56a831ed51419aecf02a89cbc7eb48de8f0e93848e8d73cf5975180502aaca1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a56a831ed51419aecf02a89cbc7eb48de8f0e93848e8d73cf5975180502aaca1/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a99672cb6ab93fd607c333290e8dacb4867c3021d76e584fa8756f7aa9385110/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a99672cb6ab93fd607c333290e8dacb4867c3021d76e584fa8756f7aa9385110/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa397f6e4634b8e2a019fa48addbf573d101c57973553024e7e833c9a2207d08/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa397f6e4634b8e2a019fa48addbf573d101c57973553024e7e833c9a2207d08/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac82ecdb883da07b23dfd96b1c3cb432ae1ea597e9e36a08d7d32f83d64d3486/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac82ecdb883da07b23dfd96b1c3cb432ae1ea597e9e36a08d7d32f83d64d3486/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad313963ad4d389ee2bd48bb11fa2d1f7b97e80459f0d44fc705931bce9194fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad313963ad4d389ee2bd48bb11fa2d1f7b97e80459f0d44fc705931bce9194fc/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b8e68337a90d4985c8d2a448ff2da30c8cd0ea586e404fe1610283e436ae38e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b8e68337a90d4985c8d2a448ff2da30c8cd0ea586e404fe1610283e436ae38e7/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bac266740cb2cba0a7f336dd16bb29f18af761b13f88a307e6df1e5f7d966c63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bac266740cb2cba0a7f336dd16bb29f18af761b13f88a307e6df1e5f7d966c63/userdata/shm major:0 minor:295 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1d4139834fa8ce4e8bab7ae6dbff1ed5ea071f1d006baa6775ed8c3b53552a7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1d4139834fa8ce4e8bab7ae6dbff1ed5ea071f1d006baa6775ed8c3b53552a7/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f71afc8fd4847c8d181bbb47535193beca676f165cb4f4aa4db71111413e4f0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f71afc8fd4847c8d181bbb47535193beca676f165cb4f4aa4db71111413e4f0d/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08817fd9-0879-4b77-a767-5f4c4ecf8d5b/volumes/kubernetes.io~projected/kube-api-access-24tfh:{mountpoint:/var/lib/kubelet/pods/08817fd9-0879-4b77-a767-5f4c4ecf8d5b/volumes/kubernetes.io~projected/kube-api-access-24tfh major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/119141fd-3c4f-4b94-9f18-6892986867e9/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/119141fd-3c4f-4b94-9f18-6892986867e9/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/119141fd-3c4f-4b94-9f18-6892986867e9/volumes/kubernetes.io~projected/kube-api-access-vkn8t:{mountpoint:/var/lib/kubelet/pods/119141fd-3c4f-4b94-9f18-6892986867e9/volumes/kubernetes.io~projected/kube-api-access-vkn8t major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/119141fd-3c4f-4b94-9f18-6892986867e9/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/119141fd-3c4f-4b94-9f18-6892986867e9/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/119141fd-3c4f-4b94-9f18-6892986867e9/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/119141fd-3c4f-4b94-9f18-6892986867e9/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1778099a-ebc0-47ba-98d0-2b0e5726e7d2/volumes/kubernetes.io~projected/kube-api-access-2kk6l:{mountpoint:/var/lib/kubelet/pods/1778099a-ebc0-47ba-98d0-2b0e5726e7d2/volumes/kubernetes.io~projected/kube-api-access-2kk6l major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19a8bde5-584f-4ff9-8e96-df33c5cda82d/volumes/kubernetes.io~projected/kube-api-access-x4qdc:{mountpoint:/var/lib/kubelet/pods/19a8bde5-584f-4ff9-8e96-df33c5cda82d/volumes/kubernetes.io~projected/kube-api-access-x4qdc major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20b9100c-2539-4227-8f2a-c93660540664/volumes/kubernetes.io~projected/kube-api-access-v2l9g:{mountpoint:/var/lib/kubelet/pods/20b9100c-2539-4227-8f2a-c93660540664/volumes/kubernetes.io~projected/kube-api-access-v2l9g major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25a7353f-76ad-4ac0-92b8-186377723478/volumes/kubernetes.io~projected/kube-api-access-7slwv:{mountpoint:/var/lib/kubelet/pods/25a7353f-76ad-4ac0-92b8-186377723478/volumes/kubernetes.io~projected/kube-api-access-7slwv major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/279b3dd8-58be-43a1-9994-59fa434409f5/volumes/kubernetes.io~projected/kube-api-access-lp9jm:{mountpoint:/var/lib/kubelet/pods/279b3dd8-58be-43a1-9994-59fa434409f5/volumes/kubernetes.io~projected/kube-api-access-lp9jm major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ed628f2-4323-47b6-ac03-b8567812a740/volumes/kubernetes.io~projected/kube-api-access-gv7pr:{mountpoint:/var/lib/kubelet/pods/2ed628f2-4323-47b6-ac03-b8567812a740/volumes/kubernetes.io~projected/kube-api-access-gv7pr major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a9e7292-b781-4a27-8a48-94f0a8843fc2/volumes/kubernetes.io~projected/kube-api-access-7k67j:{mountpoint:/var/lib/kubelet/pods/3a9e7292-b781-4a27-8a48-94f0a8843fc2/volumes/kubernetes.io~projected/kube-api-access-7k67j major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3cb721ff-442b-41b7-8287-220dbfbb41da/volumes/kubernetes.io~projected/kube-api-access-pzxps:{mountpoint:/var/lib/kubelet/pods/3cb721ff-442b-41b7-8287-220dbfbb41da/volumes/kubernetes.io~projected/kube-api-access-pzxps major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40259634-6b57-492e-8376-75ecf7900b06/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/40259634-6b57-492e-8376-75ecf7900b06/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40259634-6b57-492e-8376-75ecf7900b06/volumes/kubernetes.io~projected/kube-api-access-9t57g:{mountpoint:/var/lib/kubelet/pods/40259634-6b57-492e-8376-75ecf7900b06/volumes/kubernetes.io~projected/kube-api-access-9t57g major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40259634-6b57-492e-8376-75ecf7900b06/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/40259634-6b57-492e-8376-75ecf7900b06/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5738916f-8202-40ed-823e-90f7c9cc4ec9/volumes/kubernetes.io~projected/kube-api-access-rtmpp:{mountpoint:/var/lib/kubelet/pods/5738916f-8202-40ed-823e-90f7c9cc4ec9/volumes/kubernetes.io~projected/kube-api-access-rtmpp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58448f55-1670-41ce-8e35-182fe94f2d7e/volumes/kubernetes.io~projected/kube-api-access-bv6bs:{mountpoint:/var/lib/kubelet/pods/58448f55-1670-41ce-8e35-182fe94f2d7e/volumes/kubernetes.io~projected/kube-api-access-bv6bs major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58448f55-1670-41ce-8e35-182fe94f2d7e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/58448f55-1670-41ce-8e35-182fe94f2d7e/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66e53c53-b31d-43f7-b7b2-2c90a07dcf09/volumes/kubernetes.io~projected/kube-api-access-klntp:{mountpoint:/var/lib/kubelet/pods/66e53c53-b31d-43f7-b7b2-2c90a07dcf09/volumes/kubernetes.io~projected/kube-api-access-klntp major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6731c200-9eda-44f2-aa13-2b4a8fbda85e/volumes/kubernetes.io~projected/kube-api-access-48c5j:{mountpoint:/var/lib/kubelet/pods/6731c200-9eda-44f2-aa13-2b4a8fbda85e/volumes/kubernetes.io~projected/kube-api-access-48c5j major:0 minor:306 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/676416b5-105e-4650-baed-92baf169a9a6/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/676416b5-105e-4650-baed-92baf169a9a6/volumes/kubernetes.io~secret/agent-certs major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67ff6855-c974-451d-b582-679e7f26e98d/volumes/kubernetes.io~projected/kube-api-access-xk9fs:{mountpoint:/var/lib/kubelet/pods/67ff6855-c974-451d-b582-679e7f26e98d/volumes/kubernetes.io~projected/kube-api-access-xk9fs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78b91dbb-d9e9-4524-8827-d74c8c0625d0/volumes/kubernetes.io~projected/kube-api-access-kvnh8:{mountpoint:/var/lib/kubelet/pods/78b91dbb-d9e9-4524-8827-d74c8c0625d0/volumes/kubernetes.io~projected/kube-api-access-kvnh8 major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78b91dbb-d9e9-4524-8827-d74c8c0625d0/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/78b91dbb-d9e9-4524-8827-d74c8c0625d0/volumes/kubernetes.io~secret/cert major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79888519-e73a-4536-96e4-ce8e3ec99037/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/79888519-e73a-4536-96e4-ce8e3ec99037/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79888519-e73a-4536-96e4-ce8e3ec99037/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/79888519-e73a-4536-96e4-ce8e3ec99037/volumes/kubernetes.io~empty-dir/tmp major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79888519-e73a-4536-96e4-ce8e3ec99037/volumes/kubernetes.io~projected/kube-api-access-52hng:{mountpoint:/var/lib/kubelet/pods/79888519-e73a-4536-96e4-ce8e3ec99037/volumes/kubernetes.io~projected/kube-api-access-52hng major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e349708-a596-40ca-8724-6c2169cace8c/volumes/kubernetes.io~projected/kube-api-access-rhpl8:{mountpoint:/var/lib/kubelet/pods/7e349708-a596-40ca-8724-6c2169cace8c/volumes/kubernetes.io~projected/kube-api-access-rhpl8 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96cfe445-92c7-4bfe-bafd-8943ce48cee8/volumes/kubernetes.io~projected/kube-api-access-22cnl:{mountpoint:/var/lib/kubelet/pods/96cfe445-92c7-4bfe-bafd-8943ce48cee8/volumes/kubernetes.io~projected/kube-api-access-22cnl major:0 minor:325 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96cfe445-92c7-4bfe-bafd-8943ce48cee8/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/96cfe445-92c7-4bfe-bafd-8943ce48cee8/volumes/kubernetes.io~secret/signing-key major:0 minor:324 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd5c9498-f849-4dd6-a9e7-27a0d5db4fde/volumes/kubernetes.io~projected/kube-api-access-4q8m7:{mountpoint:/var/lib/kubelet/pods/bd5c9498-f849-4dd6-a9e7-27a0d5db4fde/volumes/kubernetes.io~projected/kube-api-access-4q8m7 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c383d3eb-10c7-4fe5-99b6-04c64472ef0a/volumes/kubernetes.io~projected/kube-api-access-nz98m:{mountpoint:/var/lib/kubelet/pods/c383d3eb-10c7-4fe5-99b6-04c64472ef0a/volumes/kubernetes.io~projected/kube-api-access-nz98m major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c620e360-6548-4b0c-ba9a-a9150cf9939c/volumes/kubernetes.io~projected/kube-api-access-sd8qs:{mountpoint:/var/lib/kubelet/pods/c620e360-6548-4b0c-ba9a-a9150cf9939c/volumes/kubernetes.io~projected/kube-api-access-sd8qs major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c620e360-6548-4b0c-ba9a-a9150cf9939c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c620e360-6548-4b0c-ba9a-a9150cf9939c/volumes/kubernetes.io~secret/serving-cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce890601-127d-4995-9487-635e7aab04f8/volumes/kubernetes.io~projected/kube-api-access-jvqfg:{mountpoint:/var/lib/kubelet/pods/ce890601-127d-4995-9487-635e7aab04f8/volumes/kubernetes.io~projected/kube-api-access-jvqfg major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce890601-127d-4995-9487-635e7aab04f8/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/ce890601-127d-4995-9487-635e7aab04f8/volumes/kubernetes.io~secret/default-certificate major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce890601-127d-4995-9487-635e7aab04f8/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/ce890601-127d-4995-9487-635e7aab04f8/volumes/kubernetes.io~secret/stats-auth major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d83648de-81dd-4160-8fa9-06d36b38d9de/volumes/kubernetes.io~projected/kube-api-access-ppt2q:{mountpoint:/var/lib/kubelet/pods/d83648de-81dd-4160-8fa9-06d36b38d9de/volumes/kubernetes.io~projected/kube-api-access-ppt2q major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d83648de-81dd-4160-8fa9-06d36b38d9de/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/d83648de-81dd-4160-8fa9-06d36b38d9de/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc162541-2ab5-4475-8856-850a7c5fd437/volumes/kubernetes.io~projected/kube-api-access-44str:{mountpoint:/var/lib/kubelet/pods/dc162541-2ab5-4475-8856-850a7c5fd437/volumes/kubernetes.io~projected/kube-api-access-44str major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df604a27-2a06-42a0-bcbd-f7a10c1420c1/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/df604a27-2a06-42a0-bcbd-f7a10c1420c1/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df604a27-2a06-42a0-bcbd-f7a10c1420c1/volumes/kubernetes.io~projected/kube-api-access-4wwkr:{mountpoint:/var/lib/kubelet/pods/df604a27-2a06-42a0-bcbd-f7a10c1420c1/volumes/kubernetes.io~projected/kube-api-access-4wwkr major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df604a27-2a06-42a0-bcbd-f7a10c1420c1/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/df604a27-2a06-42a0-bcbd-f7a10c1420c1/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df604a27-2a06-42a0-bcbd-f7a10c1420c1/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/df604a27-2a06-42a0-bcbd-f7a10c1420c1/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3/volumes/kubernetes.io~projected/kube-api-access-kg5vz:{mountpoint:/var/lib/kubelet/pods/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3/volumes/kubernetes.io~projected/kube-api-access-kg5vz major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3/volumes/kubernetes.io~secret/serving-cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/edf55612-a9f0-4fd5-ab00-ac2271acfb30/volumes/kubernetes.io~projected/kube-api-access-gvhx7:{mountpoint:/var/lib/kubelet/pods/edf55612-a9f0-4fd5-ab00-ac2271acfb30/volumes/kubernetes.io~projected/kube-api-access-gvhx7 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f142ef3a-aa21-48eb-a403-8d51d843c72d/volumes/kubernetes.io~projected/kube-api-access-8lfcx:{mountpoint:/var/lib/kubelet/pods/f142ef3a-aa21-48eb-a403-8d51d843c72d/volumes/kubernetes.io~projected/kube-api-access-8lfcx major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc09b315-54d3-44f0-b6ea-8028bd3b212b/volumes/kubernetes.io~projected/kube-api-access-9rbld:{mountpoint:/var/lib/kubelet/pods/fc09b315-54d3-44f0-b6ea-8028bd3b212b/volumes/kubernetes.io~projected/kube-api-access-9rbld major:0 minor:363 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a4270d8fb64b91e5707fa116c0744d078df42426f73a6974da706e38aade4d47/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/13c633ea42f83ad665cbd246a9bd6f20b626886e0cea0fa8f2435ee5b032cdeb/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/6be60c2f8f113d364391132db743b8aae23d8470eab057c6e2becc9aa9252e0c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/d4a5917a6df8ba521b271159356513b6f0d05f4161f98a363ae47c81779b2f04/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/adaa2c44db503a949f1fd6ec4c6bab948cce86159d907faf1e5ad23c7ddadd91/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c74cfe138fb859d911e61a6eab7b56afcdb393dce824de6cd3ae07d81f3f0576/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/733f174aa5da81d763dd09a8c1191427584a3467f545b47c6b2d407310673f2c/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/fd651a03db1307e71e6aefa2b96dcf1c28d4f3aad948fce44a040705befaa7ac/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/15ea62b9e23e8e6800b4fde708995535c4f5b0c434815e166059c50748d59a14/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/685224ce7bd4d8c145b472a52705b9e8027285691a3faae59fb13ee3c00774c7/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/8392e0a38c8e88baafdf6d5a100f8d9f48f3dd827a120bd45f86aeabf3e76d63/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/59dd1bb54e67487a6b2ba1782bd882f1cf54a0ef2cfe77934c724b7114a474c6/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/0387c481cea65b94ac1a0e848102ca59d4e59b79b34944d9421c53795937f7ca/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9d21c08e8a5a4369f8b0612295564e8b90812e048b01ce983ba1622b755f7341/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/2a127433bc24e0d353f30e5b66b3786ae51f6cba8daea93211cd2a8ffdcaa1a3/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e24e16e05d128d72aaa22961d852b0b86236ea69a2da3773b736565d57d6a47a/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/c56b88917cb8fe2794ca4dd78eb8f176d7bf0453a60e59b7cabb0d05af4e4c53/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a7c51e1967906d238700ebbf87d78655831df113181ac6424b1579f3e8881d20/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/8fd03e507be5819b5aaf0c41e902974289d3ca4f44b286054277289ebc9e38c2/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/ea1eb3c251e5767f2c68b94122a247c834ac3d10477c1ccb9eef47c7ff1e5f9c/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f51b448218792043c33a2f891d02fca10fb37373a56c0bd98d3dc4903c5a042a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/122789e593c6abfee289c967980f05aca5f6077c992b85fc1ef098b7799ea843/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/5ccad03ee8461333041a6283573923b83a109193a7d883d9612272a0020f1e1b/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/01a1ead5e21d7a78ff50feeca98130cb7112bee7be88f4fa47b32bbc685927c8/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/0f94dd1caa051ccc6216bcf2c3d95eff551d65e6d1a585077d4f05a4169a44cd/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/41c835025f3c0387a06a91631f53be2240ba6f6033517c3e0d63fa99472914ec/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/7d869292b0888f5ff3dc97f48873fcf8844ac28b66cd10ca0a637c137ef5392a/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/7e4b058ad30d9759aabe88e7b9e1b7556c976ed5edaeef4ff75815c0a6db25f5/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/21a3a95e639a08137a997b44665a13023a1ea0cd58deb83638ec38e51ff3255a/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/71444a2009cecdd6bcfb4d8e2b311b013056992690f3a7c854ca6d2b26d4ec23/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/75435184d976ba77ddee9082d34d056a46d42f8cd6374277020e17200cc70039/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/af4349dc30948b6627a8f67972eab96c9f99a6cc3ad0812e1f033921a2973114/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/f224ccd20c737d10668e59784d8c05fef8e5e66499779529ee12972b2b663974/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/cd609def55b2733ef9a4f1ec2a8d54240b3815b26375e5b2776fcf16a63ac870/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/f9934612dead8415f42ca48a0c55b4b75b82992c6ae6fd3b13012ad235adf7ae/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/4653dadbe5d893f1e130a8692e3a9aa6a1315423d5472c89495000616655eb34/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/7e4e0cb5aac3145c3c899776d1e87df726a51c131147d43bb3dadd2ccd768062/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/cc2719fbfc1e579d98cb5ee6c74afe4d10d38eda6ad0548521668719f338c30f/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/7f6cc7e781b46da84f7ab5d48064db152928117e8392e703ce070a86e917a65f/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/23554b527a207892578b5ea885e6f37d2887a58e5633b436ce13cf5ac4d96894/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/82213ffe72ed1a1ad81cfeac6f121470b3250f64bc0bc5e397c24216a4884c94/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/2c20b3f4ee8e065b80058f479c836e19d36087526b1dd5787b1cdfcfcc649894/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/1dd72aab8e3751f649a64fe42e80d1a4452eeb209a9145451795e7598fc10ace/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/f9264a61ccfe23eb8380217ab5c9d05747ff9130b4847c505d676751354b1e84/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/6a863e4bf10531f5986052f7c07068875a9b5eacb01e239e74d119bdd8da61d7/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/0be26c8f53132215ed52640745b968751f0e7d0d201203a136e9c20e2530a203/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/4f86a579e656d5e185ddbfd67b20f5376e13955071934588aba0113425c1dfd7/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/31a26b1dac89111d89e7dac7e69adc68ee1da2a90dd99b42a93a13055c553c78/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/a183ff099ac91960dbd85bb3dc5ffdc757a6c36025dcca98834cbde99350d607/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/52815f58625b3cd9328c514ed3e22d5adbf2b1b3ace224e6ed7d74c6547f09b3/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/afad74cf4674fd28c7e50e95d776a42a1943e0521067660b40d71e01308b5ca4/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/90736672e988d8c473351c64fa7398c7d3125c35bb8b8f8945b86d8c8c0935e3/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/84d02f8bfec30abf1b103b0eb6f1038e3b414b551b4ee14c107da98a38bed71b/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/b45d1a0f01c3dc7e7b90f205d9bad2809476ef054ac7700e5a814dda927dbe9a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/5410767daccbb05e1c4640c0a5b4b6f8915a05c63293401ff4200af219eac3ea/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/723c41d2de212096c9366ce177fb077292b030dfbf019b43a5dc33e24d6f216a/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/739d4efed1f52b22357393aaff69920657173e1d2403793ab3a3da626184961f/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/db25817ce8da13bbc80345f9dc3dded60bac5f04837b3f3667fdffb152bdfcd5/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/17856eae4c2ef17b6938c8c40315aafb327e684031791285726c4e3ee986a4e5/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/668a7cb07edce4971f3a2cd556acc78022f667405533329ee2e8503a6857460c/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/4abe9115c689c7fb6e65478d47063788bdf2b5b7724ed1d74bac978658f7e3e9/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/fd3ec678bb22877cd6ee4a17f601af75b84256a8711e412895294c889097939f/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 07:53:21.969099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.966088    5070 manager.go:217] Machine: {Timestamp:2026-11-03 07:53:21.965604979 +0000 UTC m=+0.080068901 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101899 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c30f8846b03117b3e346870769ecc SystemUUID:ec2c30f8-846b-0311-7b3e-346870769ecc BootID:0adaab61-516b-4804-89b7-57a4f7dc09e8 Filesystems:[{Device:/var/lib/kubelet/pods/79888519-e73a-4536-96e4-ce8e3ec99037/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a99672cb6ab93fd607c333290e8dacb4867c3021d76e584fa8756f7aa9385110/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f142ef3a-aa21-48eb-a403-8d51d843c72d/volumes/kubernetes.io~projected/kube-api-access-8lfcx DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df604a27-2a06-42a0-bcbd-f7a10c1420c1/volumes/kubernetes.io~projected/kube-api-access-4wwkr DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3cb721ff-442b-41b7-8287-220dbfbb41da/volumes/kubernetes.io~projected/kube-api-access-pzxps DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ac0604c746ebfcd6adf53fbb1d3da274c8da5401afd683fc4d5d2400398ff87/userdata/shm DeviceMajor:0 DeviceMinor:366 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58448f55-1670-41ce-8e35-182fe94f2d7e/volumes/kubernetes.io~projected/kube-api-access-bv6bs DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/119141fd-3c4f-4b94-9f18-6892986867e9/volumes/kubernetes.io~projected/kube-api-access-vkn8t DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/279b3dd8-58be-43a1-9994-59fa434409f5/volumes/kubernetes.io~projected/kube-api-access-lp9jm DeviceMajor:0 DeviceMinor:233 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42820d25b8e208745bfe099d0feb7455da75db0c9f4af218313fdf64a2c112a0/userdata/shm DeviceMajor:0 DeviceMinor:364 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/25a7353f-76ad-4ac0-92b8-186377723478/volumes/kubernetes.io~projected/kube-api-access-7slwv DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce890601-127d-4995-9487-635e7aab04f8/volumes/kubernetes.io~projected/kube-api-access-jvqfg DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3/volumes/kubernetes.io~projected/kube-api-access-kg5vz DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79888519-e73a-4536-96e4-ce8e3ec99037/volumes/kubernetes.io~projected/kube-api-access-52hng DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/96cfe445-92c7-4bfe-bafd-8943ce48cee8/volumes/kubernetes.io~projected/kube-api-access-22cnl DeviceMajor:0 DeviceMinor:325 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bd5c9498-f849-4dd6-a9e7-27a0d5db4fde/volumes/kubernetes.io~projected/kube-api-access-4q8m7 DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dc162541-2ab5-4475-8856-850a7c5fd437/volumes/kubernetes.io~projected/kube-api-access-44str DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/96cfe445-92c7-4bfe-bafd-8943ce48cee8/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:324 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/424d19eae090d708343b986bdafa6c2807fa7980e6a4e15c660c8513e9dc243b/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32284d065e9cfd2ff3d1d763bb2440817af436a19a096e3d43956a2a7429f7e8/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e702f08ed7ed20f697b4723e95f5b16c80b7b124dab15c88b47143f1709ef83/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5738916f-8202-40ed-823e-90f7c9cc4ec9/volumes/kubernetes.io~projected/kube-api-access-rtmpp DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac82ecdb883da07b23dfd96b1c3cb432ae1ea597e9e36a08d7d32f83d64d3486/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3db1086282f68b516f21655c635c9ae9a15e019a958eeb27efacbb40a9e3bf53/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df604a27-2a06-42a0-bcbd-f7a10c1420c1/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c620e360-6548-4b0c-ba9a-a9150cf9939c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c383d3eb-10c7-4fe5-99b6-04c64472ef0a/volumes/kubernetes.io~projected/kube-api-access-nz98m DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d83648de-81dd-4160-8fa9-06d36b38d9de/volumes/kubernetes.io~projected/kube-api-access-ppt2q DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/311618f3e5f892bfbf815f887a7944193e5a2914303767dc0afd89d90983ec68/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/19a8bde5-584f-4ff9-8e96-df33c5cda82d/volumes/kubernetes.io~projected/kube-api-access-x4qdc DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ce890601-127d-4995-9487-635e7aab04f8/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a9e7292-b781-4a27-8a48-94f0a8843fc2/volumes/kubernetes.io~projected/kube-api-access-7k67j DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d83648de-81dd-4160-8fa9-06d36b38d9de/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/df604a27-2a06-42a0-bcbd-f7a10c1420c1/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79888519-e73a-4536-96e4-ce8e3ec99037/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/119141fd-3c4f-4b94-9f18-6892986867e9/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/08817fd9-0879-4b77-a767-5f4c4ecf8d5b/volumes/kubernetes.io~projected/kube-api-access-24tfh DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:266 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33f8ce7f9bdc0222440db4a7bb237998b388778284b0ce351815314c1edcfd93/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/910927b265f7362ee19a8aa7142206e182ead12e4de172025dc36d0ff37c70cb/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c620e360-6548-4b0c-ba9a-a9150cf9939c/volumes/kubernetes.io~projected/kube-api-access-sd8qs DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/78b91dbb-d9e9-4524-8827-d74c8c0625d0/volumes/kubernetes.io~projected/kube-api-access-kvnh8 DeviceMajor:0 DeviceMinor:232 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/20c64d2f5beeb6f50908c2f3e422d3393f3829ca8d4366021ef2f482b4794767/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6731c200-9eda-44f2-aa13-2b4a8fbda85e/volumes/kubernetes.io~projected/kube-api-access-48c5j DeviceMajor:0 DeviceMinor:306 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa397f6e4634b8e2a019fa48addbf573d101c57973553024e7e833c9a2207d08/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/20b9100c-2539-4227-8f2a-c93660540664/volumes/kubernetes.io~projected/kube-api-access-v2l9g DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2c89b4b61002542a42a86b9447902783481924f92225ac67a1eeccf1f1af5b61/userdata/shm DeviceMajor:0 DeviceMinor:227 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ed628f2-4323-47b6-ac03-b8567812a740/volumes/kubernetes.io~projected/kube-api-access-gv7pr DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40259634-6b57-492e-8376-75ecf7900b06/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a56a831ed51419aecf02a89cbc7eb48de8f0e93848e8d73cf5975180502aaca1/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/676416b5-105e-4650-baed-92baf169a9a6/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/119141fd-3c4f-4b94-9f18-6892986867e9/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b8e68337a90d4985c8d2a448ff2da30c8cd0ea586e404fe1610283e436ae38e7/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1d4139834fa8ce4e8bab7ae6dbff1ed5ea071f1d006baa6775ed8c3b53552a7/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/40259634-6b57-492e-8376-75ecf7900b06/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7378c45cd11e438cb3bb6d12e79041ef787a8b5093548efa0943413db3b2d294/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc09b315-54d3-44f0-b6ea-8028bd3b212b/volumes/kubernetes.io~projected/kube-api-access-9rbld DeviceMajor:0 DeviceMinor:363 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/119141fd-3c4f-4b94-9f18-6892986867e9/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40259634-6b57-492e-8376-75ecf7900b06/volumes/kubernetes.io~projected/kube-api-access-9t57g DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7e349708-a596-40ca-8724-6c2169cace8c/volumes/kubernetes.io~projected/kube-api-access-rhpl8 DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/78b91dbb-d9e9-4524-8827-d74c8c0625d0/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1778099a-ebc0-47ba-98d0-2b0e5726e7d2/volumes/kubernetes.io~projected/kube-api-access-2kk6l DeviceMajor:0 DeviceMinor:358 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce890601-127d-4995-9487-635e7aab04f8/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df604a27-2a06-42a0-bcbd-f7a10c1420c1/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c0e481f1c0afebf976b20aa412a47bc61e96b6a980d3ca65fdde537b1ff35ce/userdata/shm DeviceMajor:0 DeviceMinor:316 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/33f0b316861516f8b3fea6f55511942d9d0c997927043f4ea9ed2846ec122015/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f71afc8fd4847c8d181bbb47535193beca676f165cb4f4aa4db71111413e4f0d/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58448f55-1670-41ce-8e35-182fe94f2d7e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67ff6855-c974-451d-b582-679e7f26e98d/volumes/kubernetes.io~projected/kube-api-access-xk9fs DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ead0f5b21df840f477ce3be72c0d8eb6c637190655753167aa94fc5aa524f3e/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bac266740cb2cba0a7f336dd16bb29f18af761b13f88a307e6df1e5f7d966c63/userdata/shm DeviceMajor:0 DeviceMinor:295 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/66e53c53-b31d-43f7-b7b2-2c90a07dcf09/volumes/kubernetes.io~projected/kube-api-access-klntp DeviceMajor:0 DeviceMinor:315 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad313963ad4d389ee2bd48bb11fa2d1f7b97e80459f0d44fc705931bce9194fc/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/414453077f111a09b67bb769a7b7b052ca00eeeba4a59290085399c8f9b2ceed/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f773c5bcdf6c9660c4dab8c6b494a5750da8e38e2843d8662d052cdc2b59df0/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/edf55612-a9f0-4fd5-ab00-ac2271acfb30/volumes/kubernetes.io~projected/kube-api-access-gvhx7 DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0f773c5bcdf6c96 MacAddress:9e:ef:d2:e1:3e:db Speed:10000 Mtu:8901} {Name:1ead0f5b21df840 MacAddress:9e:af:10:c2:13:bc Speed:10000 Mtu:8901} {Name:20c64d2f5beeb6f MacAddress:ae:7e:7b:fc:cb:23 Speed:10000 Mtu:8901} {Name:2ac0604c746ebfc MacAddress:d6:5e:db:a2:f2:dd Speed:10000 Mtu:8901} {Name:2c89b4b61002542 MacAddress:9e:54:c9:c6:68:6c Speed:10000 Mtu:8901} {Name:311618f3e5f892b MacAddress:52:b5:73:1f:34:bb Speed:10000 Mtu:8901} {Name:32284d065e9cfd2 MacAddress:02:f6:c5:a7:12:40 Speed:10000 Mtu:8901} {Name:33f8ce7f9bdc022 MacAddress:da:a4:22:18:25:13 Speed:10000 Mtu:8901} {Name:414453077f111a0 MacAddress:ce:04:d2:6d:79:e0 Speed:10000 Mtu:8901} {Name:42820d25b8e2087 MacAddress:42:44:aa:4c:46:00 Speed:10000 Mtu:8901} {Name:9c0e481f1c0afeb MacAddress:ea:52:c3:6f:ee:2f Speed:10000 Mtu:8901} {Name:a56a831ed51419a MacAddress:26:82:0f:3e:09:c6 Speed:10000 Mtu:8901} {Name:a99672cb6ab93fd MacAddress:1a:4f:32:95:8b:ef Speed:10000 Mtu:8901} {Name:b8e68337a90d498 MacAddress:4a:2d:8e:73:e4:06 Speed:10000 Mtu:8901} {Name:bac266740cb2cba MacAddress:a2:10:09:33:76:19 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ce:d5:e5:a9 Speed:0 Mtu:9001} {Name:br-int MacAddress:6a:33:b5:c5:04:52 Speed:0 Mtu:8901} {Name:d1d4139834fa8ce MacAddress:da:41:4c:ef:6b:8a Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ce:d5:e5:a9 Speed:0 Mtu:9001} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b2:10:87:b5:0a:55 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 07:53:21.969099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.966417    5070 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 07:53:21.969099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.966455    5070 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 07:53:21.969099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.966807    5070 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 07:53:21.969099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.966835    5070 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-123.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 07:53:21.969099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.967033    5070 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 07:53:21.969099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.967046    5070 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 07:53:21.969099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.967064    5070 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 07:53:21.969099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.967091    5070 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 07:53:21.969099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.967117    5070 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 07:53:21.969099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.967467    5070 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 07:53:21.969099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.967619    5070 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 07:53:21.969099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.967660    5070 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 07:53:21.969099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.967679    5070 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 07:53:21.969099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.967694    5070 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 07:53:21.969099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.967709    5070 kubelet.go:405] "Adding apiserver pod source"
Nov 03 07:53:21.969099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.967721    5070 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 07:53:21.969099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.968970    5070 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 07:53:21.969099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.968989    5070 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 07:53:21.970081 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.969900    5070 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 07:53:21.972112 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.972099    5070 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 07:53:21.972112 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.972111    5070 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 07:53:21.972214 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.972204    5070 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 07:53:21.972393 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.972382    5070 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 07:53:21.972423 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.972411    5070 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 07:53:21.972508 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.972499    5070 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 07:53:21.972539 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.972514    5070 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 07:53:21.972539 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.972520    5070 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 07:53:21.972539 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.972526    5070 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 07:53:21.972539 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.972532    5070 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 07:53:21.972539 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.972539    5070 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 07:53:21.972666 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.972546    5070 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 07:53:21.972666 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.972559    5070 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 07:53:21.972666 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.972567    5070 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 07:53:21.972666 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.972573    5070 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 07:53:21.972666 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.972585    5070 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 07:53:21.972666 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.972594    5070 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 07:53:21.972666 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.972620    5070 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 07:53:21.972666 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.972629    5070 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 07:53:21.976386 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.976371    5070 server.go:1263] "Started kubelet"
Nov 03 07:53:21.976495 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.976432    5070 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 07:53:21.976588 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.976545    5070 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 07:53:21.976644 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.976620    5070 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 07:53:21.976994 ip-10-0-140-123 systemd[1]: Started Kubernetes Kubelet.
Nov 03 07:53:21.977862 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.977216    5070 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 07:53:21.977862 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.977691    5070 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.982038    5070 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.982315    5070 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.982490    5070 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 07:47:42 +0000 UTC" deadline="2028-05-27 15:53:17.308622856 +0000 UTC"
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.982525    5070 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13711h59m55.326100822s"
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.982696    5070 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.982706    5070 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.982814    5070 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:21.982936    5070 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-123.us-east-2.compute.internal\" not found"
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.984271    5070 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.985461    5070 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986112    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c383d3eb-10c7-4fe5-99b6-04c64472ef0a" volumeName="kubernetes.io/projected/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-kube-api-access-nz98m" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986147    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19a8bde5-584f-4ff9-8e96-df33c5cda82d" volumeName="kubernetes.io/projected/19a8bde5-584f-4ff9-8e96-df33c5cda82d-kube-api-access-x4qdc" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986169    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ed628f2-4323-47b6-ac03-b8567812a740" volumeName="kubernetes.io/configmap/2ed628f2-4323-47b6-ac03-b8567812a740-cni-binary-copy" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986185    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c620e360-6548-4b0c-ba9a-a9150cf9939c" volumeName="kubernetes.io/empty-dir/c620e360-6548-4b0c-ba9a-a9150cf9939c-snapshots" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986203    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d83648de-81dd-4160-8fa9-06d36b38d9de" volumeName="kubernetes.io/secret/d83648de-81dd-4160-8fa9-06d36b38d9de-serving-cert" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986217    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df604a27-2a06-42a0-bcbd-f7a10c1420c1" volumeName="kubernetes.io/configmap/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-certificates" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986228    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df604a27-2a06-42a0-bcbd-f7a10c1420c1" volumeName="kubernetes.io/secret/df604a27-2a06-42a0-bcbd-f7a10c1420c1-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986244    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc09b315-54d3-44f0-b6ea-8028bd3b212b" volumeName="kubernetes.io/projected/fc09b315-54d3-44f0-b6ea-8028bd3b212b-kube-api-access-9rbld" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986262    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="119141fd-3c4f-4b94-9f18-6892986867e9" volumeName="kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-kube-api-access-vkn8t" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986280    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19a8bde5-584f-4ff9-8e96-df33c5cda82d" volumeName="kubernetes.io/configmap/19a8bde5-584f-4ff9-8e96-df33c5cda82d-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986292    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20b9100c-2539-4227-8f2a-c93660540664" volumeName="kubernetes.io/projected/20b9100c-2539-4227-8f2a-c93660540664-kube-api-access-v2l9g" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986309    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58448f55-1670-41ce-8e35-182fe94f2d7e" volumeName="kubernetes.io/configmap/58448f55-1670-41ce-8e35-182fe94f2d7e-config" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986321    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58448f55-1670-41ce-8e35-182fe94f2d7e" volumeName="kubernetes.io/secret/58448f55-1670-41ce-8e35-182fe94f2d7e-serving-cert" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986332    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c620e360-6548-4b0c-ba9a-a9150cf9939c" volumeName="kubernetes.io/projected/c620e360-6548-4b0c-ba9a-a9150cf9939c-kube-api-access-sd8qs" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986354    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce890601-127d-4995-9487-635e7aab04f8" volumeName="kubernetes.io/projected/ce890601-127d-4995-9487-635e7aab04f8-kube-api-access-jvqfg" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986366    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc162541-2ab5-4475-8856-850a7c5fd437" volumeName="kubernetes.io/projected/dc162541-2ab5-4475-8856-850a7c5fd437-kube-api-access-44str" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986383    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79888519-e73a-4536-96e4-ce8e3ec99037" volumeName="kubernetes.io/empty-dir/79888519-e73a-4536-96e4-ce8e3ec99037-etc-tuned" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986394    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd5c9498-f849-4dd6-a9e7-27a0d5db4fde" volumeName="kubernetes.io/configmap/bd5c9498-f849-4dd6-a9e7-27a0d5db4fde-iptables-alerter-script" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986411    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c620e360-6548-4b0c-ba9a-a9150cf9939c" volumeName="kubernetes.io/empty-dir/c620e360-6548-4b0c-ba9a-a9150cf9939c-tmp" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986422    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d83648de-81dd-4160-8fa9-06d36b38d9de" volumeName="kubernetes.io/configmap/d83648de-81dd-4160-8fa9-06d36b38d9de-trusted-ca" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986437    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc162541-2ab5-4475-8856-850a7c5fd437" volumeName="kubernetes.io/configmap/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-config" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986453    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3" volumeName="kubernetes.io/secret/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3-serving-cert" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986465    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08817fd9-0879-4b77-a767-5f4c4ecf8d5b" volumeName="kubernetes.io/projected/08817fd9-0879-4b77-a767-5f4c4ecf8d5b-kube-api-access-24tfh" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986482    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66e53c53-b31d-43f7-b7b2-2c90a07dcf09" volumeName="kubernetes.io/empty-dir/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-archives-path" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986494    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79888519-e73a-4536-96e4-ce8e3ec99037" volumeName="kubernetes.io/empty-dir/79888519-e73a-4536-96e4-ce8e3ec99037-tmp" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986506    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d83648de-81dd-4160-8fa9-06d36b38d9de" volumeName="kubernetes.io/projected/d83648de-81dd-4160-8fa9-06d36b38d9de-kube-api-access-ppt2q" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986522    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df604a27-2a06-42a0-bcbd-f7a10c1420c1" volumeName="kubernetes.io/empty-dir/df604a27-2a06-42a0-bcbd-f7a10c1420c1-ca-trust-extracted" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986541    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5738916f-8202-40ed-823e-90f7c9cc4ec9" volumeName="kubernetes.io/projected/5738916f-8202-40ed-823e-90f7c9cc4ec9-kube-api-access-rtmpp" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986569    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67ff6855-c974-451d-b582-679e7f26e98d" volumeName="kubernetes.io/projected/67ff6855-c974-451d-b582-679e7f26e98d-kube-api-access-xk9fs" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986582    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df604a27-2a06-42a0-bcbd-f7a10c1420c1" volumeName="kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-bound-sa-token" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986598    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40259634-6b57-492e-8376-75ecf7900b06" volumeName="kubernetes.io/configmap/40259634-6b57-492e-8376-75ecf7900b06-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986964    5070 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.986610    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="676416b5-105e-4650-baed-92baf169a9a6" volumeName="kubernetes.io/configmap/676416b5-105e-4650-baed-92baf169a9a6-konnectivity-ca" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987165    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78b91dbb-d9e9-4524-8827-d74c8c0625d0" volumeName="kubernetes.io/secret/78b91dbb-d9e9-4524-8827-d74c8c0625d0-cert" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987180    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c383d3eb-10c7-4fe5-99b6-04c64472ef0a" volumeName="kubernetes.io/configmap/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-telemetry-config" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987191    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df604a27-2a06-42a0-bcbd-f7a10c1420c1" volumeName="kubernetes.io/secret/df604a27-2a06-42a0-bcbd-f7a10c1420c1-installation-pull-secrets" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987208    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="119141fd-3c4f-4b94-9f18-6892986867e9" volumeName="kubernetes.io/secret/119141fd-3c4f-4b94-9f18-6892986867e9-image-registry-private-configuration" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987220    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ed628f2-4323-47b6-ac03-b8567812a740" volumeName="kubernetes.io/configmap/2ed628f2-4323-47b6-ac03-b8567812a740-multus-daemon-config" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987237    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="119141fd-3c4f-4b94-9f18-6892986867e9" volumeName="kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-bound-sa-token" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987250    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3cb721ff-442b-41b7-8287-220dbfbb41da" volumeName="kubernetes.io/projected/3cb721ff-442b-41b7-8287-220dbfbb41da-kube-api-access-pzxps" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987277    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40259634-6b57-492e-8376-75ecf7900b06" volumeName="kubernetes.io/configmap/40259634-6b57-492e-8376-75ecf7900b06-ovnkube-config" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987290    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6731c200-9eda-44f2-aa13-2b4a8fbda85e" volumeName="kubernetes.io/projected/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-api-access-48c5j" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987302    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="676416b5-105e-4650-baed-92baf169a9a6" volumeName="kubernetes.io/secret/676416b5-105e-4650-baed-92baf169a9a6-agent-certs" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987322    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c620e360-6548-4b0c-ba9a-a9150cf9939c" volumeName="kubernetes.io/secret/c620e360-6548-4b0c-ba9a-a9150cf9939c-serving-cert" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987334    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce890601-127d-4995-9487-635e7aab04f8" volumeName="kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-stats-auth" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987349    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3" volumeName="kubernetes.io/configmap/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3-config" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987362    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="119141fd-3c4f-4b94-9f18-6892986867e9" volumeName="kubernetes.io/configmap/119141fd-3c4f-4b94-9f18-6892986867e9-trusted-ca" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987373    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="119141fd-3c4f-4b94-9f18-6892986867e9" volumeName="kubernetes.io/secret/119141fd-3c4f-4b94-9f18-6892986867e9-installation-pull-secrets" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987389    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="279b3dd8-58be-43a1-9994-59fa434409f5" volumeName="kubernetes.io/configmap/279b3dd8-58be-43a1-9994-59fa434409f5-config-volume" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987420    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66e53c53-b31d-43f7-b7b2-2c90a07dcf09" volumeName="kubernetes.io/configmap/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987438    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd5c9498-f849-4dd6-a9e7-27a0d5db4fde" volumeName="kubernetes.io/projected/bd5c9498-f849-4dd6-a9e7-27a0d5db4fde-kube-api-access-4q8m7" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987452    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df604a27-2a06-42a0-bcbd-f7a10c1420c1" volumeName="kubernetes.io/configmap/df604a27-2a06-42a0-bcbd-f7a10c1420c1-trusted-ca" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987467    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3" volumeName="kubernetes.io/projected/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3-kube-api-access-kg5vz" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987487    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edf55612-a9f0-4fd5-ab00-ac2271acfb30" volumeName="kubernetes.io/projected/edf55612-a9f0-4fd5-ab00-ac2271acfb30-kube-api-access-gvhx7" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987500    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1778099a-ebc0-47ba-98d0-2b0e5726e7d2" volumeName="kubernetes.io/projected/1778099a-ebc0-47ba-98d0-2b0e5726e7d2-kube-api-access-2kk6l" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987518    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="751205b0-8312-4ea6-920c-9517e19566f2" volumeName="kubernetes.io/configmap/751205b0-8312-4ea6-920c-9517e19566f2-nginx-conf" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987532    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d83648de-81dd-4160-8fa9-06d36b38d9de" volumeName="kubernetes.io/configmap/d83648de-81dd-4160-8fa9-06d36b38d9de-config" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987570    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df" volumeName="kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987589    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40259634-6b57-492e-8376-75ecf7900b06" volumeName="kubernetes.io/secret/40259634-6b57-492e-8376-75ecf7900b06-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987600    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66e53c53-b31d-43f7-b7b2-2c90a07dcf09" volumeName="kubernetes.io/configmap/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-service-ca-bundle" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987612    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79888519-e73a-4536-96e4-ce8e3ec99037" volumeName="kubernetes.io/projected/79888519-e73a-4536-96e4-ce8e3ec99037-kube-api-access-52hng" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987629    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25a7353f-76ad-4ac0-92b8-186377723478" volumeName="kubernetes.io/projected/25a7353f-76ad-4ac0-92b8-186377723478-kube-api-access-7slwv" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987642    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40259634-6b57-492e-8376-75ecf7900b06" volumeName="kubernetes.io/configmap/40259634-6b57-492e-8376-75ecf7900b06-env-overrides" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987659    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6731c200-9eda-44f2-aa13-2b4a8fbda85e" volumeName="kubernetes.io/empty-dir/6731c200-9eda-44f2-aa13-2b4a8fbda85e-data-volume" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987675    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96cfe445-92c7-4bfe-bafd-8943ce48cee8" volumeName="kubernetes.io/projected/96cfe445-92c7-4bfe-bafd-8943ce48cee8-kube-api-access-22cnl" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987689    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96cfe445-92c7-4bfe-bafd-8943ce48cee8" volumeName="kubernetes.io/secret/96cfe445-92c7-4bfe-bafd-8943ce48cee8-signing-key" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987847    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f142ef3a-aa21-48eb-a403-8d51d843c72d" volumeName="kubernetes.io/projected/f142ef3a-aa21-48eb-a403-8d51d843c72d-kube-api-access-8lfcx" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987868    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19a8bde5-584f-4ff9-8e96-df33c5cda82d" volumeName="kubernetes.io/configmap/19a8bde5-584f-4ff9-8e96-df33c5cda82d-cni-binary-copy" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987884    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19a8bde5-584f-4ff9-8e96-df33c5cda82d" volumeName="kubernetes.io/configmap/19a8bde5-584f-4ff9-8e96-df33c5cda82d-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987901    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="279b3dd8-58be-43a1-9994-59fa434409f5" volumeName="kubernetes.io/empty-dir/279b3dd8-58be-43a1-9994-59fa434409f5-tmp-dir" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987921    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58448f55-1670-41ce-8e35-182fe94f2d7e" volumeName="kubernetes.io/projected/58448f55-1670-41ce-8e35-182fe94f2d7e-kube-api-access-bv6bs" seLinuxMountContext=""
Nov 03 07:53:21.987852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987934    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66e53c53-b31d-43f7-b7b2-2c90a07dcf09" volumeName="kubernetes.io/projected/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-kube-api-access-klntp" seLinuxMountContext=""
Nov 03 07:53:21.990614 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987967    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96cfe445-92c7-4bfe-bafd-8943ce48cee8" volumeName="kubernetes.io/configmap/96cfe445-92c7-4bfe-bafd-8943ce48cee8-signing-cabundle" seLinuxMountContext=""
Nov 03 07:53:21.990614 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987979    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c620e360-6548-4b0c-ba9a-a9150cf9939c" volumeName="kubernetes.io/configmap/c620e360-6548-4b0c-ba9a-a9150cf9939c-service-ca-bundle" seLinuxMountContext=""
Nov 03 07:53:21.990614 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.987987    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce890601-127d-4995-9487-635e7aab04f8" volumeName="kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-default-certificate" seLinuxMountContext=""
Nov 03 07:53:21.990614 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.988038    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78b91dbb-d9e9-4524-8827-d74c8c0625d0" volumeName="kubernetes.io/projected/78b91dbb-d9e9-4524-8827-d74c8c0625d0-kube-api-access-kvnh8" seLinuxMountContext=""
Nov 03 07:53:21.990614 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.988059    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c620e360-6548-4b0c-ba9a-a9150cf9939c" volumeName="kubernetes.io/configmap/c620e360-6548-4b0c-ba9a-a9150cf9939c-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 07:53:21.990614 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.988075    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="279b3dd8-58be-43a1-9994-59fa434409f5" volumeName="kubernetes.io/projected/279b3dd8-58be-43a1-9994-59fa434409f5-kube-api-access-lp9jm" seLinuxMountContext=""
Nov 03 07:53:21.990614 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.988093    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ed628f2-4323-47b6-ac03-b8567812a740" volumeName="kubernetes.io/projected/2ed628f2-4323-47b6-ac03-b8567812a740-kube-api-access-gv7pr" seLinuxMountContext=""
Nov 03 07:53:21.990614 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.988105    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a9e7292-b781-4a27-8a48-94f0a8843fc2" volumeName="kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j" seLinuxMountContext=""
Nov 03 07:53:21.990614 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.988126    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e349708-a596-40ca-8724-6c2169cace8c" volumeName="kubernetes.io/projected/7e349708-a596-40ca-8724-6c2169cace8c-kube-api-access-rhpl8" seLinuxMountContext=""
Nov 03 07:53:21.990614 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.988140    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f142ef3a-aa21-48eb-a403-8d51d843c72d" volumeName="kubernetes.io/empty-dir/f142ef3a-aa21-48eb-a403-8d51d843c72d-tmp-dir" seLinuxMountContext=""
Nov 03 07:53:21.990614 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.988151    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40259634-6b57-492e-8376-75ecf7900b06" volumeName="kubernetes.io/projected/40259634-6b57-492e-8376-75ecf7900b06-kube-api-access-9t57g" seLinuxMountContext=""
Nov 03 07:53:21.990614 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.988168    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="119141fd-3c4f-4b94-9f18-6892986867e9" volumeName="kubernetes.io/configmap/119141fd-3c4f-4b94-9f18-6892986867e9-registry-certificates" seLinuxMountContext=""
Nov 03 07:53:21.990614 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.988184    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="119141fd-3c4f-4b94-9f18-6892986867e9" volumeName="kubernetes.io/empty-dir/119141fd-3c4f-4b94-9f18-6892986867e9-ca-trust-extracted" seLinuxMountContext=""
Nov 03 07:53:21.990614 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.988202    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67ff6855-c974-451d-b582-679e7f26e98d" volumeName="kubernetes.io/configmap/67ff6855-c974-451d-b582-679e7f26e98d-serviceca" seLinuxMountContext=""
Nov 03 07:53:21.990614 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.988220    5070 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df604a27-2a06-42a0-bcbd-f7a10c1420c1" volumeName="kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-kube-api-access-4wwkr" seLinuxMountContext=""
Nov 03 07:53:21.990614 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.988232    5070 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 07:53:21.990614 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.988239    5070 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 07:53:21.990614 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.990575    5070 factory.go:55] Registering systemd factory
Nov 03 07:53:21.990614 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.990590    5070 factory.go:223] Registration of the systemd container factory successfully
Nov 03 07:53:21.991467 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.990877    5070 factory.go:153] Registering CRI-O factory
Nov 03 07:53:21.991467 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.990892    5070 factory.go:223] Registration of the crio container factory successfully
Nov 03 07:53:21.991467 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.990944    5070 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 07:53:21.991467 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.990968    5070 factory.go:103] Registering Raw factory
Nov 03 07:53:21.991467 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.990981    5070 manager.go:1199] Started watching for new ooms in manager
Nov 03 07:53:21.991467 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.991337    5070 manager.go:319] Starting recovery of all containers
Nov 03 07:53:21.993523 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:21.993432    5070 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 07:53:22.004615 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.004585    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/1.log"
Nov 03 07:53:22.004947 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.004916    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/1.log"
Nov 03 07:53:22.015180 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.015147    5070 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 07:53:22.016871 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.016853    5070 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 07:53:22.016976 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.016966    5070 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 07:53:22.017072 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.017054    5070 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 07:53:22.017155 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:22.017095    5070 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 07:53:22.019643 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.019625    5070 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 07:53:22.025130 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.025105    5070 generic.go:356] "Generic (PLEG): container finished" podID="3012e3a7d17ffd2dd3688b2234e0123b" containerID="99692de9354655f075332cd7bb12593f7b6658aea489d064d59ce3c6cd4ffcd2" exitCode=0
Nov 03 07:53:22.037175 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.037150    5070 generic.go:356] "Generic (PLEG): container finished" podID="19a8bde5-584f-4ff9-8e96-df33c5cda82d" containerID="59b9f2e73298e3a7f55a96eedc7ef2eb5c680ab4bcebf6f2d95faab060a8555e" exitCode=0
Nov 03 07:53:22.037259 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.037176    5070 generic.go:356] "Generic (PLEG): container finished" podID="19a8bde5-584f-4ff9-8e96-df33c5cda82d" containerID="0ba8f3f03e7da096ab4560cbc38cc3b90e9e07de7c166bb790acc38c5bf3bc8a" exitCode=0
Nov 03 07:53:22.037259 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.037187    5070 generic.go:356] "Generic (PLEG): container finished" podID="19a8bde5-584f-4ff9-8e96-df33c5cda82d" containerID="c8a43580ef3f3303a88b4206fa559f3ba85baea124abe36d73b6b07f00678a48" exitCode=0
Nov 03 07:53:22.037259 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.037198    5070 generic.go:356] "Generic (PLEG): container finished" podID="19a8bde5-584f-4ff9-8e96-df33c5cda82d" containerID="f544ec45dc8b0ed1730be896018a9530650d5de2a403da0bda714d6c28019c39" exitCode=0
Nov 03 07:53:22.037259 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.037208    5070 generic.go:356] "Generic (PLEG): container finished" podID="19a8bde5-584f-4ff9-8e96-df33c5cda82d" containerID="7c7962c466399ed8a199e725facfbbaeb7216957516a59264a707dae3e5783a9" exitCode=0
Nov 03 07:53:22.037259 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.037218    5070 generic.go:356] "Generic (PLEG): container finished" podID="19a8bde5-584f-4ff9-8e96-df33c5cda82d" containerID="fd7003fa158b4562f41e538a4a9a79512b3747e95a33da3c6e26e2a9f5b4966a" exitCode=0
Nov 03 07:53:22.072901 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.072881    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/1.log"
Nov 03 07:53:22.072987 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.072907    5070 generic.go:356] "Generic (PLEG): container finished" podID="d83648de-81dd-4160-8fa9-06d36b38d9de" containerID="b17a031689eca024fee8f71239c507cd7dc10ca5e0147ff1e9b4d8440b1d8575" exitCode=255
Nov 03 07:53:22.096173 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.096128    5070 manager.go:324] Recovery completed
Nov 03 07:53:22.116826 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.116807    5070 cpu_manager.go:226] "Starting" policy="none"
Nov 03 07:53:22.116826 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.116822    5070 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 07:53:22.116975 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.116842    5070 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 07:53:22.117041 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.116983    5070 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 07:53:22.117097 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.117033    5070 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 07:53:22.117097 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.117067    5070 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 07:53:22.117097 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.117076    5070 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 07:53:22.117097 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.117084    5070 policy_none.go:50] "Start"
Nov 03 07:53:22.117097 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.117096    5070 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 07:53:22.117328 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.117112    5070 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 07:53:22.117328 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.117288    5070 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 07:53:22.117328 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.117299    5070 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 07:53:22.117328 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.117308    5070 policy_none.go:44] "Start"
Nov 03 07:53:22.117544 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:22.117470    5070 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 07:53:22.121719 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.121694    5070 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 07:53:22.125140 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.125128    5070 manager.go:341] "Starting Device Plugin manager"
Nov 03 07:53:22.125597 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:22.125581    5070 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 07:53:22.125597 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.125597    5070 server.go:87] "Starting device plugin registration server"
Nov 03 07:53:22.125846 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.125835    5070 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 07:53:22.125891 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.125846    5070 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 07:53:22.126115 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.125942    5070 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 07:53:22.126115 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.126028    5070 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 07:53:22.126115 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.126064    5070 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 07:53:22.126115 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.126071    5070 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 07:53:22.126273 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.126134    5070 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T07:53:22.126053175Z","UUID":"c379c541-6d11-4aed-912a-d5fcbdf66242","Handler":null,"Name":"","Endpoint":""}
Nov 03 07:53:22.127906 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:22.127874    5070 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 07:53:22.128216 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.128196    5070 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 07:53:22.128277 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.128232    5070 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 07:53:22.226785 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.226749    5070 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 07:53:22.227862 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.227848    5070 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Nov 03 07:53:22.227933 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.227876    5070 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 03 07:53:22.227933 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.227886    5070 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-123.us-east-2.compute.internal" event="NodeHasSufficientPID"
Nov 03 07:53:22.227933 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.227923    5070 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:53:22.237346 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.237330    5070 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:53:22.237418 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.237387    5070 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:53:22.318054 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.318012    5070 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal"]
Nov 03 07:53:22.318527 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.318498    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal" event={"ID":"3012e3a7d17ffd2dd3688b2234e0123b","Type":"ContainerStarted","Data":"48146d79563f9d6c4482160a29a06f7f0badfa26caf9e8c2d3c108bb5fad1c60"}
Nov 03 07:53:22.318609 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.318533    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal" event={"ID":"3012e3a7d17ffd2dd3688b2234e0123b","Type":"ContainerDied","Data":"99692de9354655f075332cd7bb12593f7b6658aea489d064d59ce3c6cd4ffcd2"}
Nov 03 07:53:22.318609 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.318548    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal" event={"ID":"3012e3a7d17ffd2dd3688b2234e0123b","Type":"ContainerStarted","Data":"ad313963ad4d389ee2bd48bb11fa2d1f7b97e80459f0d44fc705931bce9194fc"}
Nov 03 07:53:22.318609 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.318574    5070 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:53:22.318609 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.318587    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal" event={"ID":"fc5342d3d37d4243b5710bd6f9020da4","Type":"ContainerStarted","Data":"ae5476d431be031cbd30539a8848a33f604db2815ec69d0c5a52b42d245db564"}
Nov 03 07:53:22.318609 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.318599    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal" event={"ID":"fc5342d3d37d4243b5710bd6f9020da4","Type":"ContainerStarted","Data":"1e702f08ed7ed20f697b4723e95f5b16c80b7b124dab15c88b47143f1709ef83"}
Nov 03 07:53:22.318861 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.318805    5070 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:53:22.330165 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.330147    5070 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 07:53:22.330225 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:22.330179    5070 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:53:22.330608 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.330593    5070 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 07:53:22.330642 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:22.330627    5070 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:53:22.394927 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.394903    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3012e3a7d17ffd2dd3688b2234e0123b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal\" (UID: \"3012e3a7d17ffd2dd3688b2234e0123b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:53:22.395013 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.394934    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc5342d3d37d4243b5710bd6f9020da4-config\") pod \"kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal\" (UID: \"fc5342d3d37d4243b5710bd6f9020da4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:53:22.395013 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.394951    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3012e3a7d17ffd2dd3688b2234e0123b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal\" (UID: \"3012e3a7d17ffd2dd3688b2234e0123b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:53:22.495500 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.495476    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc5342d3d37d4243b5710bd6f9020da4-config\") pod \"kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal\" (UID: \"fc5342d3d37d4243b5710bd6f9020da4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:53:22.495574 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.495502    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3012e3a7d17ffd2dd3688b2234e0123b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal\" (UID: \"3012e3a7d17ffd2dd3688b2234e0123b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:53:22.495574 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.495519    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3012e3a7d17ffd2dd3688b2234e0123b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal\" (UID: \"3012e3a7d17ffd2dd3688b2234e0123b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:53:22.495574 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.495561    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3012e3a7d17ffd2dd3688b2234e0123b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal\" (UID: \"3012e3a7d17ffd2dd3688b2234e0123b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:53:22.495683 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.495585    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3012e3a7d17ffd2dd3688b2234e0123b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal\" (UID: \"3012e3a7d17ffd2dd3688b2234e0123b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:53:22.495683 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.495591    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc5342d3d37d4243b5710bd6f9020da4-config\") pod \"kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal\" (UID: \"fc5342d3d37d4243b5710bd6f9020da4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:53:22.968865 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.968843    5070 apiserver.go:52] "Watching apiserver"
Nov 03 07:53:22.974534 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.974514    5070 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 07:53:22.975423 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.975398    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-kj8jw","openshift-dns/dns-default-46l5d","kube-system/kas-connection-checker-685895c7c9-kr2q8","openshift-dns/node-resolver-drdft","openshift-insights/insights-operator-7c4986d449-crn8p","openshift-multus/multus-additional-cni-plugins-h4dgx","openshift-ingress-canary/ingress-canary-6ldkm","openshift-multus/network-metrics-daemon-x7cmt","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zhh6g","openshift-multus/multus-msq9p","kube-system/kas-connection-checker-685895c7c9-nwtkn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db","openshift-cluster-node-tuning-operator/tuned-fgwf4","kube-system/kas-connection-checker-685895c7c9-v2z4k","kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal","openshift-insights/insights-runtime-extractor-wd497","openshift-network-diagnostics/network-check-source-6f78d59f99-fshz5","openshift-console-operator/console-operator-7cd56b8dbc-5qpnb","kube-system/global-pull-secret-syncer-ljbh2","openshift-insights/periodic-gathering-l5f6q-fpwdg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal","openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks","kube-system/konnectivity-agent-9j2f9","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2","openshift-image-registry/image-registry-67ffc88546-z24dr","openshift-kube-storage-version-migrator/migrator-66df5d496b-fhz8r","openshift-ingress/router-default-79698c9d55-m2lbh","openshift-network-console/networking-console-plugin-86c478bb88-grgjj","openshift-kube-storage-version-migrator/migrator-66df5d496b-d7zfg","openshift-image-registry/node-ca-w9c2l","openshift-image-registry/image-registry-74774885c5-97jxz","openshift-network-diagnostics/network-check-target-c6bms","openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f","openshift-ovn-kubernetes/ovnkube-node-x6d47","openshift-service-ca/service-ca-5fffc97bdf-cnskv","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk"]
Nov 03 07:53:22.979467 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.979447    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:22.980303 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.980283    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:22.981416 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.981399    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:22.981959 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.981941    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Nov 03 07:53:22.981959 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.981940    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Nov 03 07:53:22.982072 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.982047    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Nov 03 07:53:22.982263 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.982244    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Nov 03 07:53:22.982542 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.982523    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Nov 03 07:53:22.982628 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.982620    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 07:53:22.982685 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.982636    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 07:53:22.982738 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.982692    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 07:53:22.983266 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.983241    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:22.983834 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.983818    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-grgjj"
Nov 03 07:53:22.984293 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.984253    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Nov 03 07:53:22.984430 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.984349    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Nov 03 07:53:22.984430 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.984372    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Nov 03 07:53:22.984430 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.984419    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Nov 03 07:53:22.984646 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.984484    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Nov 03 07:53:22.984646 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.984495    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Nov 03 07:53:22.984646 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.984543    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 07:53:22.984646 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.984596    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Nov 03 07:53:22.984974 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.984695    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Nov 03 07:53:22.984974 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.984819    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 07:53:22.984974 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.984837    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 07:53:22.984974 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.984875    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 07:53:22.984974 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.984874    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Nov 03 07:53:22.984974 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.984905    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Nov 03 07:53:22.984974 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.984915    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Nov 03 07:53:22.985499 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.985000    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 07:53:22.985499 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.985016    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Nov 03 07:53:22.985499 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.985042    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Nov 03 07:53:22.985499 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.985057    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:53:22.985499 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.985097    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Nov 03 07:53:22.985499 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.985293    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Nov 03 07:53:22.985499 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.985322    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Nov 03 07:53:22.985499 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.985358    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 07:53:22.985499 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.985457    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Nov 03 07:53:22.985499 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.985464    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Nov 03 07:53:22.985499 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.985475    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 07:53:22.986093 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.985961    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Nov 03 07:53:22.986684 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.986663    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 07:53:22.986684 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.986680    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Nov 03 07:53:22.986684 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.986685    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 07:53:22.987243 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.986734    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Nov 03 07:53:22.987243 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.986794    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 07:53:22.987243 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.986793    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 07:53:22.987243 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.987054    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-cg8rh\""
Nov 03 07:53:22.987243 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.987136    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Nov 03 07:53:22.987243 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.987149    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 07:53:22.987750 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.987351    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Nov 03 07:53:22.987926 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.987815    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 07:53:22.987926 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.987833    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 07:53:22.988320 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.988303    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 07:53:22.989545 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.989523    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 07:53:22.989742 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.989717    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k9glt\""
Nov 03 07:53:22.989825 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.989742    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 07:53:22.989874 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.989830    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-drdft_f142ef3a-aa21-48eb-a403-8d51d843c72d/dns-node-resolver/0.log"
Nov 03 07:53:22.990042 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.990001    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 07:53:22.990042 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.990030    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 07:53:22.990404 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.990388    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 07:53:22.990710 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.990695    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Nov 03 07:53:22.991086 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.991067    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 07:53:22.991180 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.991124    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 07:53:22.991636 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.991612    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dl4bp\""
Nov 03 07:53:22.991713 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.991696    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2mmmx\""
Nov 03 07:53:22.993230 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.993210    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 07:53:22.994117 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.994095    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 07:53:22.994344 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.994329    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dc6md\""
Nov 03 07:53:22.994540 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.994345    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 07:53:22.994540 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.994432    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 07:53:22.994540 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.994481    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5b8js\""
Nov 03 07:53:22.994540 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.994427    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fpwxk\""
Nov 03 07:53:22.995838 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.994935    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 07:53:22.995838 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.994936    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 07:53:22.995838 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.995215    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 07:53:22.995838 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.995220    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-hq49v\""
Nov 03 07:53:22.995838 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.995233    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 07:53:22.997485 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.997461    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Nov 03 07:53:22.997772 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.997742    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:22.997845 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.997789    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk\" (UID: \"e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk"
Nov 03 07:53:22.997899 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.997840    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhpl8\" (UniqueName: \"kubernetes.io/projected/7e349708-a596-40ca-8724-6c2169cace8c-kube-api-access-rhpl8\") pod \"network-check-source-6f78d59f99-fshz5\" (UID: \"7e349708-a596-40ca-8724-6c2169cace8c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fshz5"
Nov 03 07:53:22.997947 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.997933    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-certificates\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:22.998005 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.997993    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 07:53:22.998061 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998000    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df604a27-2a06-42a0-bcbd-f7a10c1420c1-trusted-ca\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:22.998139 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998056    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 07:53:22.998139 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998063    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk\" (UID: \"e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk"
Nov 03 07:53:22.998139 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998056    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d83648de-81dd-4160-8fa9-06d36b38d9de-config\") pod \"console-operator-7cd56b8dbc-5qpnb\" (UID: \"d83648de-81dd-4160-8fa9-06d36b38d9de\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:22.998139 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998112    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 07:53:22.998347 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998147    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzxps\" (UniqueName: \"kubernetes.io/projected/3cb721ff-442b-41b7-8287-220dbfbb41da-kube-api-access-pzxps\") pod \"volume-data-source-validator-58c9f5f46d-zhh6g\" (UID: \"3cb721ff-442b-41b7-8287-220dbfbb41da\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zhh6g"
Nov 03 07:53:22.998347 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998168    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df604a27-2a06-42a0-bcbd-f7a10c1420c1-ca-trust-extracted\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:22.998347 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998178    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bmmvq\""
Nov 03 07:53:22.998347 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998187    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2l9g\" (UniqueName: \"kubernetes.io/projected/20b9100c-2539-4227-8f2a-c93660540664-kube-api-access-v2l9g\") pod \"kas-connection-checker-685895c7c9-v2z4k\" (UID: \"20b9100c-2539-4227-8f2a-c93660540664\") " pod="kube-system/kas-connection-checker-685895c7c9-v2z4k"
Nov 03 07:53:22.998347 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998210    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dtgx2\" (UID: \"dc162541-2ab5-4475-8856-850a7c5fd437\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:22.998347 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998268    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c620e360-6548-4b0c-ba9a-a9150cf9939c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:22.998347 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998293    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c620e360-6548-4b0c-ba9a-a9150cf9939c-service-ca-bundle\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:22.998347 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998308    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d83648de-81dd-4160-8fa9-06d36b38d9de-config\") pod \"console-operator-7cd56b8dbc-5qpnb\" (UID: \"d83648de-81dd-4160-8fa9-06d36b38d9de\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:22.998347 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998317    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-f68ks\" (UID: \"c383d3eb-10c7-4fe5-99b6-04c64472ef0a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:22.998347 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998343    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvqfg\" (UniqueName: \"kubernetes.io/projected/ce890601-127d-4995-9487-635e7aab04f8-kube-api-access-jvqfg\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:22.998893 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998367    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:22.998893 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998390    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wwkr\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-kube-api-access-4wwkr\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:22.998893 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998415    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz98m\" (UniqueName: \"kubernetes.io/projected/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-kube-api-access-nz98m\") pod \"cluster-monitoring-operator-884576c57-f68ks\" (UID: \"c383d3eb-10c7-4fe5-99b6-04c64472ef0a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:22.998893 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998439    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppt2q\" (UniqueName: \"kubernetes.io/projected/d83648de-81dd-4160-8fa9-06d36b38d9de-kube-api-access-ppt2q\") pod \"console-operator-7cd56b8dbc-5qpnb\" (UID: \"d83648de-81dd-4160-8fa9-06d36b38d9de\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:22.998893 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998463    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv6bs\" (UniqueName: \"kubernetes.io/projected/58448f55-1670-41ce-8e35-182fe94f2d7e-kube-api-access-bv6bs\") pod \"service-ca-operator-544d749c8c-gt59f\" (UID: \"58448f55-1670-41ce-8e35-182fe94f2d7e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f"
Nov 03 07:53:22.998893 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998577    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df604a27-2a06-42a0-bcbd-f7a10c1420c1-trusted-ca\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:22.998893 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998650    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df604a27-2a06-42a0-bcbd-f7a10c1420c1-ca-trust-extracted\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:22.998893 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998827    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-certificates\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:22.998893 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998854    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c620e360-6548-4b0c-ba9a-a9150cf9939c-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:22.998893 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998869    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df604a27-2a06-42a0-bcbd-f7a10c1420c1-image-registry-private-configuration\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:22.999433 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998912    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd8qs\" (UniqueName: \"kubernetes.io/projected/c620e360-6548-4b0c-ba9a-a9150cf9939c-kube-api-access-sd8qs\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:22.999433 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998946    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-metrics-certs\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:22.999433 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998973    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d83648de-81dd-4160-8fa9-06d36b38d9de-serving-cert\") pod \"console-operator-7cd56b8dbc-5qpnb\" (UID: \"d83648de-81dd-4160-8fa9-06d36b38d9de\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:22.999433 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.998998    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk\" (UID: \"e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk"
Nov 03 07:53:22.999433 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999023    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg5vz\" (UniqueName: \"kubernetes.io/projected/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3-kube-api-access-kg5vz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk\" (UID: \"e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk"
Nov 03 07:53:22.999433 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999061    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df604a27-2a06-42a0-bcbd-f7a10c1420c1-image-registry-private-configuration\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:22.999433 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999174    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d83648de-81dd-4160-8fa9-06d36b38d9de-serving-cert\") pod \"console-operator-7cd56b8dbc-5qpnb\" (UID: \"d83648de-81dd-4160-8fa9-06d36b38d9de\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:22.999433 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999222    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk\" (UID: \"e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk"
Nov 03 07:53:22.999433 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999231    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-bound-sa-token\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:22.999433 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999273    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dtgx2\" (UID: \"dc162541-2ab5-4475-8856-850a7c5fd437\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:22.999433 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999301    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-f68ks\" (UID: \"c383d3eb-10c7-4fe5-99b6-04c64472ef0a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:22.999433 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999327    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d83648de-81dd-4160-8fa9-06d36b38d9de-trusted-ca\") pod \"console-operator-7cd56b8dbc-5qpnb\" (UID: \"d83648de-81dd-4160-8fa9-06d36b38d9de\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:22.999433 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999352    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/58448f55-1670-41ce-8e35-182fe94f2d7e-config\") pod \"service-ca-operator-544d749c8c-gt59f\" (UID: \"58448f55-1670-41ce-8e35-182fe94f2d7e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f"
Nov 03 07:53:22.999433 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999377    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-default-certificate\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:22.999433 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999419    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-stats-auth\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:23.000164 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999439    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c620e360-6548-4b0c-ba9a-a9150cf9939c-service-ca-bundle\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:23.000164 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999446    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24tfh\" (UniqueName: \"kubernetes.io/projected/08817fd9-0879-4b77-a767-5f4c4ecf8d5b-kube-api-access-24tfh\") pod \"kas-connection-checker-685895c7c9-nwtkn\" (UID: \"08817fd9-0879-4b77-a767-5f4c4ecf8d5b\") " pod="kube-system/kas-connection-checker-685895c7c9-nwtkn"
Nov 03 07:53:23.000164 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999527    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58448f55-1670-41ce-8e35-182fe94f2d7e-serving-cert\") pod \"service-ca-operator-544d749c8c-gt59f\" (UID: \"58448f55-1670-41ce-8e35-182fe94f2d7e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f"
Nov 03 07:53:23.000164 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999558    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df604a27-2a06-42a0-bcbd-f7a10c1420c1-installation-pull-secrets\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:23.000164 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999589    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c620e360-6548-4b0c-ba9a-a9150cf9939c-tmp\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:23.000164 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999614    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c620e360-6548-4b0c-ba9a-a9150cf9939c-snapshots\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:23.000164 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999659    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c620e360-6548-4b0c-ba9a-a9150cf9939c-serving-cert\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:23.000164 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999675    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 07:53:23.000164 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999687    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvhx7\" (UniqueName: \"kubernetes.io/projected/edf55612-a9f0-4fd5-ab00-ac2271acfb30-kube-api-access-gvhx7\") pod \"kas-connection-checker-685895c7c9-kr2q8\" (UID: \"edf55612-a9f0-4fd5-ab00-ac2271acfb30\") " pod="kube-system/kas-connection-checker-685895c7c9-kr2q8"
Nov 03 07:53:23.000164 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999713    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44str\" (UniqueName: \"kubernetes.io/projected/dc162541-2ab5-4475-8856-850a7c5fd437-kube-api-access-44str\") pod \"cluster-samples-operator-6ddb6c8b94-dtgx2\" (UID: \"dc162541-2ab5-4475-8856-850a7c5fd437\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:23.000164 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999741    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-thqm7\""
Nov 03 07:53:23.000164 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999884    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/58448f55-1670-41ce-8e35-182fe94f2d7e-config\") pod \"service-ca-operator-544d749c8c-gt59f\" (UID: \"58448f55-1670-41ce-8e35-182fe94f2d7e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f"
Nov 03 07:53:23.000164 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.000004    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/58448f55-1670-41ce-8e35-182fe94f2d7e-serving-cert\") pod \"service-ca-operator-544d749c8c-gt59f\" (UID: \"58448f55-1670-41ce-8e35-182fe94f2d7e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f"
Nov 03 07:53:23.000164 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.000094    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-f68ks\" (UID: \"c383d3eb-10c7-4fe5-99b6-04c64472ef0a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:23.000164 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.000127    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-default-certificate\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:23.000164 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.000165    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c620e360-6548-4b0c-ba9a-a9150cf9939c-tmp\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:23.000688 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.000180    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d83648de-81dd-4160-8fa9-06d36b38d9de-trusted-ca\") pod \"console-operator-7cd56b8dbc-5qpnb\" (UID: \"d83648de-81dd-4160-8fa9-06d36b38d9de\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:23.000688 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.000194    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df604a27-2a06-42a0-bcbd-f7a10c1420c1-installation-pull-secrets\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:23.000688 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:22.999562    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-dtgx2\" (UID: \"dc162541-2ab5-4475-8856-850a7c5fd437\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:23.000688 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.000276    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c620e360-6548-4b0c-ba9a-a9150cf9939c-snapshots\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:23.000688 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.000291    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c620e360-6548-4b0c-ba9a-a9150cf9939c-serving-cert\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:23.000688 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.000332    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-stats-auth\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:23.000688 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.000449    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 07:53:23.000923 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.000902    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:23.001991 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.001963    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-c2t62\""
Nov 03 07:53:23.001991 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.001975    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 07:53:23.002100 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.002087    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 07:53:23.002184 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.002172    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkwf4\""
Nov 03 07:53:23.002313 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.002300    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:23.002506 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.002489    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 07:53:23.002948 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.002934    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:23.003733 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.003712    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2l9g\" (UniqueName: \"kubernetes.io/projected/20b9100c-2539-4227-8f2a-c93660540664-kube-api-access-v2l9g\") pod \"kas-connection-checker-685895c7c9-v2z4k\" (UID: \"20b9100c-2539-4227-8f2a-c93660540664\") " pod="kube-system/kas-connection-checker-685895c7c9-v2z4k"
Nov 03 07:53:23.003897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.003865    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzxps\" (UniqueName: \"kubernetes.io/projected/3cb721ff-442b-41b7-8287-220dbfbb41da-kube-api-access-pzxps\") pod \"volume-data-source-validator-58c9f5f46d-zhh6g\" (UID: \"3cb721ff-442b-41b7-8287-220dbfbb41da\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-zhh6g"
Nov 03 07:53:23.004008 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.003969    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhpl8\" (UniqueName: \"kubernetes.io/projected/7e349708-a596-40ca-8724-6c2169cace8c-kube-api-access-rhpl8\") pod \"network-check-source-6f78d59f99-fshz5\" (UID: \"7e349708-a596-40ca-8724-6c2169cace8c\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-fshz5"
Nov 03 07:53:23.004164 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.004147    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppt2q\" (UniqueName: \"kubernetes.io/projected/d83648de-81dd-4160-8fa9-06d36b38d9de-kube-api-access-ppt2q\") pod \"console-operator-7cd56b8dbc-5qpnb\" (UID: \"d83648de-81dd-4160-8fa9-06d36b38d9de\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:23.004526 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.004506    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv6bs\" (UniqueName: \"kubernetes.io/projected/58448f55-1670-41ce-8e35-182fe94f2d7e-kube-api-access-bv6bs\") pod \"service-ca-operator-544d749c8c-gt59f\" (UID: \"58448f55-1670-41ce-8e35-182fe94f2d7e\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f"
Nov 03 07:53:23.004710 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.004672    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 07:53:23.005047 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.005032    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 07:53:23.005133 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.005056    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 07:53:23.005191 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.005180    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 07:53:23.005261 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.005242    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 07:53:23.005477 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.005460    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 07:53:23.005544 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.005496    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg5vz\" (UniqueName: \"kubernetes.io/projected/e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3-kube-api-access-kg5vz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk\" (UID: \"e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk"
Nov 03 07:53:23.005886 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.005863    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz98m\" (UniqueName: \"kubernetes.io/projected/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-kube-api-access-nz98m\") pod \"cluster-monitoring-operator-884576c57-f68ks\" (UID: \"c383d3eb-10c7-4fe5-99b6-04c64472ef0a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:23.006225 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.006206    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd8qs\" (UniqueName: \"kubernetes.io/projected/c620e360-6548-4b0c-ba9a-a9150cf9939c-kube-api-access-sd8qs\") pod \"insights-operator-7c4986d449-crn8p\" (UID: \"c620e360-6548-4b0c-ba9a-a9150cf9939c\") " pod="openshift-insights/insights-operator-7c4986d449-crn8p"
Nov 03 07:53:23.006386 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.006370    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 07:53:23.006444 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.006408    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvqfg\" (UniqueName: \"kubernetes.io/projected/ce890601-127d-4995-9487-635e7aab04f8-kube-api-access-jvqfg\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:23.006493 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.006414    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24tfh\" (UniqueName: \"kubernetes.io/projected/08817fd9-0879-4b77-a767-5f4c4ecf8d5b-kube-api-access-24tfh\") pod \"kas-connection-checker-685895c7c9-nwtkn\" (UID: \"08817fd9-0879-4b77-a767-5f4c4ecf8d5b\") " pod="kube-system/kas-connection-checker-685895c7c9-nwtkn"
Nov 03 07:53:23.006556 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.006513    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wwkr\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-kube-api-access-4wwkr\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:23.006744 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.006727    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-bound-sa-token\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:23.006886 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.006869    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvhx7\" (UniqueName: \"kubernetes.io/projected/edf55612-a9f0-4fd5-ab00-ac2271acfb30-kube-api-access-gvhx7\") pod \"kas-connection-checker-685895c7c9-kr2q8\" (UID: \"edf55612-a9f0-4fd5-ab00-ac2271acfb30\") " pod="kube-system/kas-connection-checker-685895c7c9-kr2q8"
Nov 03 07:53:23.007189 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.007155    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 07:53:23.007383 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.007372    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 07:53:23.008194 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.008180    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44str\" (UniqueName: \"kubernetes.io/projected/dc162541-2ab5-4475-8856-850a7c5fd437-kube-api-access-44str\") pod \"cluster-samples-operator-6ddb6c8b94-dtgx2\" (UID: \"dc162541-2ab5-4475-8856-850a7c5fd437\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:23.011800 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.011785    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 07:53:23.011880 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.011867    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 07:53:23.059677 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.059654    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 07:53:23.060280 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.060257    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 07:53:23.060370 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.060259    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 07:53:23.060370 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.060287    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-tpfjw\""
Nov 03 07:53:23.060370 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.060315    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rmlwc\""
Nov 03 07:53:23.060370 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.060350    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m4xzk\""
Nov 03 07:53:23.060370 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.060357    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ghck6\""
Nov 03 07:53:23.060556 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.060403    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-28mc2\""
Nov 03 07:53:23.060556 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.060408    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zjbhz\""
Nov 03 07:53:23.060556 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.060412    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 07:53:23.060556 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.060465    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vbzhs\""
Nov 03 07:53:23.060556 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.060505    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5spf6\""
Nov 03 07:53:23.082302 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.082286    5070 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:53:23.084074 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.084058    5070 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 07:53:23.089742 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.089728    5070 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 07:53:23.089817 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:23.089784    5070 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal"
Nov 03 07:53:23.100209 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.100191    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-sys-fs\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:53:23.100249 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.100217    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk9fs\" (UniqueName: \"kubernetes.io/projected/67ff6855-c974-451d-b582-679e7f26e98d-kube-api-access-xk9fs\") pod \"node-ca-w9c2l\" (UID: \"67ff6855-c974-451d-b582-679e7f26e98d\") " pod="openshift-image-registry/node-ca-w9c2l"
Nov 03 07:53:23.100249 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.100236    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4qdc\" (UniqueName: \"kubernetes.io/projected/19a8bde5-584f-4ff9-8e96-df33c5cda82d-kube-api-access-x4qdc\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:53:23.100303 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.100251    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-etc-selinux\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:53:23.100303 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.100267    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtmpp\" (UniqueName: \"kubernetes.io/projected/5738916f-8202-40ed-823e-90f7c9cc4ec9-kube-api-access-rtmpp\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:53:23.100303 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.100292    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-run\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.100538 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.100521    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-var-lib-kubelet\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.100569 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.100559    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2ed628f2-4323-47b6-ac03-b8567812a740-multus-daemon-config\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.100637 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.100624    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv7pr\" (UniqueName: \"kubernetes.io/projected/2ed628f2-4323-47b6-ac03-b8567812a740-kube-api-access-gv7pr\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.100777 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.100737    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-run-netns\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.100817 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.100787    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-sys\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.100873 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.100858    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:23.100903 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.100892    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/96cfe445-92c7-4bfe-bafd-8943ce48cee8-signing-key\") pod \"service-ca-5fffc97bdf-cnskv\" (UID: \"96cfe445-92c7-4bfe-bafd-8943ce48cee8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cnskv"
Nov 03 07:53:23.100903 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.100895    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2ed628f2-4323-47b6-ac03-b8567812a740-multus-daemon-config\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.100961 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.100908    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rbld\" (UniqueName: \"kubernetes.io/projected/fc09b315-54d3-44f0-b6ea-8028bd3b212b-kube-api-access-9rbld\") pod \"migrator-66df5d496b-d7zfg\" (UID: \"fc09b315-54d3-44f0-b6ea-8028bd3b212b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-d7zfg"
Nov 03 07:53:23.100961 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.100926    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79888519-e73a-4536-96e4-ce8e3ec99037-etc-tuned\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.100961 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.100954    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q8m7\" (UniqueName: \"kubernetes.io/projected/bd5c9498-f849-4dd6-a9e7-27a0d5db4fde-kube-api-access-4q8m7\") pod \"iptables-alerter-kj8jw\" (UID: \"bd5c9498-f849-4dd6-a9e7-27a0d5db4fde\") " pod="openshift-network-operator/iptables-alerter-kj8jw"
Nov 03 07:53:23.101072 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.100982    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19a8bde5-584f-4ff9-8e96-df33c5cda82d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:53:23.101072 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:23.101018    5070 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle podName:ce890601-127d-4995-9487-635e7aab04f8 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:23.60099712 +0000 UTC m=+1.715461059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle") pod "router-default-79698c9d55-m2lbh" (UID: "ce890601-127d-4995-9487-635e7aab04f8") : configmap references non-existent config key: service-ca.crt
Nov 03 07:53:23.101072 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101032    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/79888519-e73a-4536-96e4-ce8e3ec99037-etc-tuned\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.101072 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101041    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lfcx\" (UniqueName: \"kubernetes.io/projected/f142ef3a-aa21-48eb-a403-8d51d843c72d-kube-api-access-8lfcx\") pod \"node-resolver-drdft\" (UID: \"f142ef3a-aa21-48eb-a403-8d51d843c72d\") " pod="openshift-dns/node-resolver-drdft"
Nov 03 07:53:23.101271 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101152    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/279b3dd8-58be-43a1-9994-59fa434409f5-config-volume\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:23.101271 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101170    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/96cfe445-92c7-4bfe-bafd-8943ce48cee8-signing-key\") pod \"service-ca-5fffc97bdf-cnskv\" (UID: \"96cfe445-92c7-4bfe-bafd-8943ce48cee8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cnskv"
Nov 03 07:53:23.101271 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101178    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/40259634-6b57-492e-8376-75ecf7900b06-ovn-node-metrics-cert\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.101271 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101212    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k67j\" (UniqueName: \"kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j\") pod \"network-check-target-c6bms\" (UID: \"3a9e7292-b781-4a27-8a48-94f0a8843fc2\") " pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:53:23.101271 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101240    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-device-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:53:23.101271 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101264    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79888519-e73a-4536-96e4-ce8e3ec99037-tmp\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.101526 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101305    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19a8bde5-584f-4ff9-8e96-df33c5cda82d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:53:23.101526 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101313    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6731c200-9eda-44f2-aa13-2b4a8fbda85e-crio-socket\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:23.101526 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101336    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79888519-e73a-4536-96e4-ce8e3ec99037-tmp\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.101526 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101338    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19a8bde5-584f-4ff9-8e96-df33c5cda82d-cnibin\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:53:23.101526 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101349    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/279b3dd8-58be-43a1-9994-59fa434409f5-config-volume\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:23.101526 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101374    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-var-lib-openvswitch\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.101526 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101398    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd5c9498-f849-4dd6-a9e7-27a0d5db4fde-iptables-alerter-script\") pod \"iptables-alerter-kj8jw\" (UID: \"bd5c9498-f849-4dd6-a9e7-27a0d5db4fde\") " pod="openshift-network-operator/iptables-alerter-kj8jw"
Nov 03 07:53:23.101526 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101425    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/119141fd-3c4f-4b94-9f18-6892986867e9-ca-trust-extracted\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:23.101526 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101470    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-archives-path\") pod \"periodic-gathering-l5f6q-fpwdg\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") " pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:23.101526 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101503    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f142ef3a-aa21-48eb-a403-8d51d843c72d-tmp-dir\") pod \"node-resolver-drdft\" (UID: \"f142ef3a-aa21-48eb-a403-8d51d843c72d\") " pod="openshift-dns/node-resolver-drdft"
Nov 03 07:53:23.101977 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101529    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-archives-path\") pod \"periodic-gathering-l5f6q-fpwdg\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") " pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:23.101977 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101538    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/40259634-6b57-492e-8376-75ecf7900b06-ovn-node-metrics-cert\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.101977 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101530    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.101977 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101475    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/119141fd-3c4f-4b94-9f18-6892986867e9-ca-trust-extracted\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:23.101977 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101603    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f142ef3a-aa21-48eb-a403-8d51d843c72d-tmp-dir\") pod \"node-resolver-drdft\" (UID: \"f142ef3a-aa21-48eb-a403-8d51d843c72d\") " pod="openshift-dns/node-resolver-drdft"
Nov 03 07:53:23.101977 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101647    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t57g\" (UniqueName: \"kubernetes.io/projected/40259634-6b57-492e-8376-75ecf7900b06-kube-api-access-9t57g\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.101977 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101669    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd5c9498-f849-4dd6-a9e7-27a0d5db4fde-iptables-alerter-script\") pod \"iptables-alerter-kj8jw\" (UID: \"bd5c9498-f849-4dd6-a9e7-27a0d5db4fde\") " pod="openshift-network-operator/iptables-alerter-kj8jw"
Nov 03 07:53:23.101977 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101680    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/119141fd-3c4f-4b94-9f18-6892986867e9-image-registry-private-configuration\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:23.101977 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101696    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/119141fd-3c4f-4b94-9f18-6892986867e9-trusted-ca\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:23.101977 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101711    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-sysctl-d\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.101977 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101726    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52hng\" (UniqueName: \"kubernetes.io/projected/79888519-e73a-4536-96e4-ce8e3ec99037-kube-api-access-52hng\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.101977 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101740    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-run-k8s-cni-cncf-io\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.101977 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101777    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19a8bde5-584f-4ff9-8e96-df33c5cda82d-cni-binary-copy\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:53:23.101977 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101802    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-run-systemd\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.101977 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101831    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-systemd\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.101977 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101855    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-bound-sa-token\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:23.101977 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101860    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/119141fd-3c4f-4b94-9f18-6892986867e9-image-registry-private-configuration\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:23.101977 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101898    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-service-ca-bundle\") pod \"periodic-gathering-l5f6q-fpwdg\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") " pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:23.101977 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101939    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f142ef3a-aa21-48eb-a403-8d51d843c72d-hosts-file\") pod \"node-resolver-drdft\" (UID: \"f142ef3a-aa21-48eb-a403-8d51d843c72d\") " pod="openshift-dns/node-resolver-drdft"
Nov 03 07:53:23.101977 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.101986    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-dbus\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:53:23.102897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.102032    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19a8bde5-584f-4ff9-8e96-df33c5cda82d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:53:23.102897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.102053    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/119141fd-3c4f-4b94-9f18-6892986867e9-trusted-ca\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:23.102897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.102074    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78b91dbb-d9e9-4524-8827-d74c8c0625d0-cert\") pod \"ingress-canary-6ldkm\" (UID: \"78b91dbb-d9e9-4524-8827-d74c8c0625d0\") " pod="openshift-ingress-canary/ingress-canary-6ldkm"
Nov 03 07:53:23.102897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.102131    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/40259634-6b57-492e-8376-75ecf7900b06-ovnkube-script-lib\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.102897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.102162    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19a8bde5-584f-4ff9-8e96-df33c5cda82d-cni-binary-copy\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:53:23.102897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.102201    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkn8t\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-kube-api-access-vkn8t\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:23.102897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.102312    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/78b91dbb-d9e9-4524-8827-d74c8c0625d0-cert\") pod \"ingress-canary-6ldkm\" (UID: \"78b91dbb-d9e9-4524-8827-d74c8c0625d0\") " pod="openshift-ingress-canary/ingress-canary-6ldkm"
Nov 03 07:53:23.102897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.102336    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/40259634-6b57-492e-8376-75ecf7900b06-ovnkube-script-lib\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.102897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.102345    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs\") pod \"network-metrics-daemon-x7cmt\" (UID: \"25a7353f-76ad-4ac0-92b8-186377723478\") " pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:53:23.102897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.102399    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-metrics-certs\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:23.102897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.102437    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-var-lib-cni-bin\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.102897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.102436    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-service-ca-bundle\") pod \"periodic-gathering-l5f6q-fpwdg\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") " pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:23.102897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.102539    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48c5j\" (UniqueName: \"kubernetes.io/projected/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-api-access-48c5j\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:23.102897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.102586    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-run-ovn\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.102897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.102617    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-node-log\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.102897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.102667    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279b3dd8-58be-43a1-9994-59fa434409f5-tmp-dir\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:23.102897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.102699    5070 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 07:53:23.102897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.102830    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/40259634-6b57-492e-8376-75ecf7900b06-ovnkube-config\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.102897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.102874    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279b3dd8-58be-43a1-9994-59fa434409f5-tmp-dir\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:23.102897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.102913    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103006    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-system-cni-dir\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103054    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-multus-cni-dir\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103100    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-sysconfig\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103130    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-sysctl-conf\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103157    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/40259634-6b57-492e-8376-75ecf7900b06-ovnkube-config\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103159    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67ff6855-c974-451d-b582-679e7f26e98d-host\") pod \"node-ca-w9c2l\" (UID: \"67ff6855-c974-451d-b582-679e7f26e98d\") " pod="openshift-image-registry/node-ca-w9c2l"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103186    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-log-socket\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103211    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-run-ovn-kubernetes\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103249    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-registration-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103288    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/751205b0-8312-4ea6-920c-9517e19566f2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-grgjj\" (UID: \"751205b0-8312-4ea6-920c-9517e19566f2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grgjj"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103317    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22cnl\" (UniqueName: \"kubernetes.io/projected/96cfe445-92c7-4bfe-bafd-8943ce48cee8-kube-api-access-22cnl\") pod \"service-ca-5fffc97bdf-cnskv\" (UID: \"96cfe445-92c7-4bfe-bafd-8943ce48cee8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cnskv"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103342    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103409    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-kubelet\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103447    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19a8bde5-584f-4ff9-8e96-df33c5cda82d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103484    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-systemd-units\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103495    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/751205b0-8312-4ea6-920c-9517e19566f2-nginx-conf\") pod \"networking-console-plugin-86c478bb88-grgjj\" (UID: \"751205b0-8312-4ea6-920c-9517e19566f2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grgjj"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103568    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-cni-netd\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103604    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-trusted-ca-bundle\") pod \"periodic-gathering-l5f6q-fpwdg\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") " pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103626    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-modprobe-d\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103641    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19a8bde5-584f-4ff9-8e96-df33c5cda82d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103645    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/751205b0-8312-4ea6-920c-9517e19566f2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-grgjj\" (UID: \"751205b0-8312-4ea6-920c-9517e19566f2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grgjj"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103688    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-etc-openvswitch\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103719    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:23.103749 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103749    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67ff6855-c974-451d-b582-679e7f26e98d-serviceca\") pod \"node-ca-w9c2l\" (UID: \"67ff6855-c974-451d-b582-679e7f26e98d\") " pod="openshift-image-registry/node-ca-w9c2l"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103801    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-cnibin\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103876    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ed628f2-4323-47b6-ac03-b8567812a740-cni-binary-copy\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103885    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-trusted-ca-bundle\") pod \"periodic-gathering-l5f6q-fpwdg\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") " pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103924    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-multus-socket-dir-parent\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103957    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-kubelet-config\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.103993    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-slash\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104039    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104078    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ed628f2-4323-47b6-ac03-b8567812a740-cni-binary-copy\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104080    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klntp\" (UniqueName: \"kubernetes.io/projected/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-kube-api-access-klntp\") pod \"periodic-gathering-l5f6q-fpwdg\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") " pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104125    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/96cfe445-92c7-4bfe-bafd-8943ce48cee8-signing-cabundle\") pod \"service-ca-5fffc97bdf-cnskv\" (UID: \"96cfe445-92c7-4bfe-bafd-8943ce48cee8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cnskv"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104181    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-run-netns\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104288    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-var-lib-kubelet\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104322    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-var-lib-cni-multus\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104327    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-original-pull-secret\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104333    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/96cfe445-92c7-4bfe-bafd-8943ce48cee8-signing-cabundle\") pod \"service-ca-5fffc97bdf-cnskv\" (UID: \"96cfe445-92c7-4bfe-bafd-8943ce48cee8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cnskv"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104362    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-hostroot\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104392    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-run-multus-certs\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104408    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/67ff6855-c974-451d-b582-679e7f26e98d-serviceca\") pod \"node-ca-w9c2l\" (UID: \"67ff6855-c974-451d-b582-679e7f26e98d\") " pod="openshift-image-registry/node-ca-w9c2l"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104432    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-socket-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104460    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-kubernetes\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104497    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-lib-modules\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104525    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/279b3dd8-58be-43a1-9994-59fa434409f5-metrics-tls\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:23.104555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104541    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-etc-kubernetes\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104586    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/119141fd-3c4f-4b94-9f18-6892986867e9-registry-certificates\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104617    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/676416b5-105e-4650-baed-92baf169a9a6-konnectivity-ca\") pod \"konnectivity-agent-9j2f9\" (UID: \"676416b5-105e-4650-baed-92baf169a9a6\") " pod="kube-system/konnectivity-agent-9j2f9"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104653    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp9jm\" (UniqueName: \"kubernetes.io/projected/279b3dd8-58be-43a1-9994-59fa434409f5-kube-api-access-lp9jm\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104679    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7slwv\" (UniqueName: \"kubernetes.io/projected/25a7353f-76ad-4ac0-92b8-186377723478-kube-api-access-7slwv\") pod \"network-metrics-daemon-x7cmt\" (UID: \"25a7353f-76ad-4ac0-92b8-186377723478\") " pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104811    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6731c200-9eda-44f2-aa13-2b4a8fbda85e-data-volume\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104821    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/676416b5-105e-4650-baed-92baf169a9a6-konnectivity-ca\") pod \"konnectivity-agent-9j2f9\" (UID: \"676416b5-105e-4650-baed-92baf169a9a6\") " pod="kube-system/konnectivity-agent-9j2f9"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104918    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-run-openvswitch\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104957    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6731c200-9eda-44f2-aa13-2b4a8fbda85e-data-volume\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.104966    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-cni-bin\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.105014    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dtgx2\" (UID: \"dc162541-2ab5-4475-8856-850a7c5fd437\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.105051    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-f68ks\" (UID: \"c383d3eb-10c7-4fe5-99b6-04c64472ef0a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.105070    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/676416b5-105e-4650-baed-92baf169a9a6-agent-certs\") pod \"konnectivity-agent-9j2f9\" (UID: \"676416b5-105e-4650-baed-92baf169a9a6\") " pod="kube-system/konnectivity-agent-9j2f9"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.105085    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvnh8\" (UniqueName: \"kubernetes.io/projected/78b91dbb-d9e9-4524-8827-d74c8c0625d0-kube-api-access-kvnh8\") pod \"ingress-canary-6ldkm\" (UID: \"78b91dbb-d9e9-4524-8827-d74c8c0625d0\") " pod="openshift-ingress-canary/ingress-canary-6ldkm"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.105109    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/40259634-6b57-492e-8376-75ecf7900b06-env-overrides\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.105133    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.105161    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.105189    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19a8bde5-584f-4ff9-8e96-df33c5cda82d-system-cni-dir\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.105214    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd5c9498-f849-4dd6-a9e7-27a0d5db4fde-host-slash\") pod \"iptables-alerter-kj8jw\" (UID: \"bd5c9498-f849-4dd6-a9e7-27a0d5db4fde\") " pod="openshift-network-operator/iptables-alerter-kj8jw"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.105241    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/119141fd-3c4f-4b94-9f18-6892986867e9-installation-pull-secrets\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.105267    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kk6l\" (UniqueName: \"kubernetes.io/projected/1778099a-ebc0-47ba-98d0-2b0e5726e7d2-kube-api-access-2kk6l\") pod \"migrator-66df5d496b-fhz8r\" (UID: \"1778099a-ebc0-47ba-98d0-2b0e5726e7d2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fhz8r"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.105292    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-host\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.105322    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-multus-conf-dir\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.105383    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/676416b5-105e-4650-baed-92baf169a9a6-agent-certs\") pod \"konnectivity-agent-9j2f9\" (UID: \"676416b5-105e-4650-baed-92baf169a9a6\") " pod="kube-system/konnectivity-agent-9j2f9"
Nov 03 07:53:23.105496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.105396    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/119141fd-3c4f-4b94-9f18-6892986867e9-registry-certificates\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:23.106338 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.105610    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/119141fd-3c4f-4b94-9f18-6892986867e9-installation-pull-secrets\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:23.106338 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.105651    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/40259634-6b57-492e-8376-75ecf7900b06-env-overrides\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.106338 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:23.105748    5070 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 07:53:23.106338 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:23.105787    5070 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74774885c5-97jxz: secret "image-registry-tls" not found
Nov 03 07:53:23.106338 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:23.105868    5070 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls podName:df604a27-2a06-42a0-bcbd-f7a10c1420c1 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:23.605825802 +0000 UTC m=+1.720289714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls") pod "image-registry-74774885c5-97jxz" (UID: "df604a27-2a06-42a0-bcbd-f7a10c1420c1") : secret "image-registry-tls" not found
Nov 03 07:53:23.106846 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.106826    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce890601-127d-4995-9487-635e7aab04f8-metrics-certs\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:23.107403 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.107383    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc162541-2ab5-4475-8856-850a7c5fd437-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-dtgx2\" (UID: \"dc162541-2ab5-4475-8856-850a7c5fd437\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:23.107482 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.107450    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c383d3eb-10c7-4fe5-99b6-04c64472ef0a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-f68ks\" (UID: \"c383d3eb-10c7-4fe5-99b6-04c64472ef0a\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:23.107999 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.107963    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k67j\" (UniqueName: \"kubernetes.io/projected/3a9e7292-b781-4a27-8a48-94f0a8843fc2-kube-api-access-7k67j\") pod \"network-check-target-c6bms\" (UID: \"3a9e7292-b781-4a27-8a48-94f0a8843fc2\") " pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:53:23.108039 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.108021    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtmpp\" (UniqueName: \"kubernetes.io/projected/5738916f-8202-40ed-823e-90f7c9cc4ec9-kube-api-access-rtmpp\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:53:23.108123 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.108105    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk9fs\" (UniqueName: \"kubernetes.io/projected/67ff6855-c974-451d-b582-679e7f26e98d-kube-api-access-xk9fs\") pod \"node-ca-w9c2l\" (UID: \"67ff6855-c974-451d-b582-679e7f26e98d\") " pod="openshift-image-registry/node-ca-w9c2l"
Nov 03 07:53:23.108637 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.108612    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4qdc\" (UniqueName: \"kubernetes.io/projected/19a8bde5-584f-4ff9-8e96-df33c5cda82d-kube-api-access-x4qdc\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:53:23.108720 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.108646    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q8m7\" (UniqueName: \"kubernetes.io/projected/bd5c9498-f849-4dd6-a9e7-27a0d5db4fde-kube-api-access-4q8m7\") pod \"iptables-alerter-kj8jw\" (UID: \"bd5c9498-f849-4dd6-a9e7-27a0d5db4fde\") " pod="openshift-network-operator/iptables-alerter-kj8jw"
Nov 03 07:53:23.108720 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.108710    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lfcx\" (UniqueName: \"kubernetes.io/projected/f142ef3a-aa21-48eb-a403-8d51d843c72d-kube-api-access-8lfcx\") pod \"node-resolver-drdft\" (UID: \"f142ef3a-aa21-48eb-a403-8d51d843c72d\") " pod="openshift-dns/node-resolver-drdft"
Nov 03 07:53:23.108828 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.108808    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rbld\" (UniqueName: \"kubernetes.io/projected/fc09b315-54d3-44f0-b6ea-8028bd3b212b-kube-api-access-9rbld\") pod \"migrator-66df5d496b-d7zfg\" (UID: \"fc09b315-54d3-44f0-b6ea-8028bd3b212b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-d7zfg"
Nov 03 07:53:23.109029 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.109006    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t57g\" (UniqueName: \"kubernetes.io/projected/40259634-6b57-492e-8376-75ecf7900b06-kube-api-access-9t57g\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.109141 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.109049    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkn8t\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-kube-api-access-vkn8t\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:23.109320 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.109300    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv7pr\" (UniqueName: \"kubernetes.io/projected/2ed628f2-4323-47b6-ac03-b8567812a740-kube-api-access-gv7pr\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.109573 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.109556    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48c5j\" (UniqueName: \"kubernetes.io/projected/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-api-access-48c5j\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:23.109648 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.109619    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klntp\" (UniqueName: \"kubernetes.io/projected/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-kube-api-access-klntp\") pod \"periodic-gathering-l5f6q-fpwdg\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") " pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:53:23.109714 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.109662    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52hng\" (UniqueName: \"kubernetes.io/projected/79888519-e73a-4536-96e4-ce8e3ec99037-kube-api-access-52hng\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.109924 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.109907    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-bound-sa-token\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:23.109972 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.109922    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22cnl\" (UniqueName: \"kubernetes.io/projected/96cfe445-92c7-4bfe-bafd-8943ce48cee8-kube-api-access-22cnl\") pod \"service-ca-5fffc97bdf-cnskv\" (UID: \"96cfe445-92c7-4bfe-bafd-8943ce48cee8\") " pod="openshift-service-ca/service-ca-5fffc97bdf-cnskv"
Nov 03 07:53:23.113582 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.113558    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kk6l\" (UniqueName: \"kubernetes.io/projected/1778099a-ebc0-47ba-98d0-2b0e5726e7d2-kube-api-access-2kk6l\") pod \"migrator-66df5d496b-fhz8r\" (UID: \"1778099a-ebc0-47ba-98d0-2b0e5726e7d2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fhz8r"
Nov 03 07:53:23.113693 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.113678    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp9jm\" (UniqueName: \"kubernetes.io/projected/279b3dd8-58be-43a1-9994-59fa434409f5-kube-api-access-lp9jm\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:23.114495 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.114479    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7slwv\" (UniqueName: \"kubernetes.io/projected/25a7353f-76ad-4ac0-92b8-186377723478-kube-api-access-7slwv\") pod \"network-metrics-daemon-x7cmt\" (UID: \"25a7353f-76ad-4ac0-92b8-186377723478\") " pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:53:23.114669 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.114653    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvnh8\" (UniqueName: \"kubernetes.io/projected/78b91dbb-d9e9-4524-8827-d74c8c0625d0-kube-api-access-kvnh8\") pod \"ingress-canary-6ldkm\" (UID: \"78b91dbb-d9e9-4524-8827-d74c8c0625d0\") " pod="openshift-ingress-canary/ingress-canary-6ldkm"
Nov 03 07:53:23.206340 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206321    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-device-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:53:23.206436 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206346    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6731c200-9eda-44f2-aa13-2b4a8fbda85e-crio-socket\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:23.206436 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206361    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19a8bde5-584f-4ff9-8e96-df33c5cda82d-cnibin\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:53:23.206436 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206382    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-var-lib-openvswitch\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.206436 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206406    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.206591 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206450    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-device-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:53:23.206591 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206452    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6731c200-9eda-44f2-aa13-2b4a8fbda85e-crio-socket\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:23.206591 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206464    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-var-lib-openvswitch\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.206591 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206458    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19a8bde5-584f-4ff9-8e96-df33c5cda82d-cnibin\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:53:23.206591 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206475    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-sysctl-d\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.206591 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206451    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.206591 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206552    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-run-k8s-cni-cncf-io\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.206591 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206569    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-run-systemd\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.206591 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206571    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-sysctl-d\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.206591 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206583    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-systemd\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206600    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-run-k8s-cni-cncf-io\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206606    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f142ef3a-aa21-48eb-a403-8d51d843c72d-hosts-file\") pod \"node-resolver-drdft\" (UID: \"f142ef3a-aa21-48eb-a403-8d51d843c72d\") " pod="openshift-dns/node-resolver-drdft"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206629    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-dbus\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206643    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-systemd\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206652    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19a8bde5-584f-4ff9-8e96-df33c5cda82d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206670    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f142ef3a-aa21-48eb-a403-8d51d843c72d-hosts-file\") pod \"node-resolver-drdft\" (UID: \"f142ef3a-aa21-48eb-a403-8d51d843c72d\") " pod="openshift-dns/node-resolver-drdft"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206678    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs\") pod \"network-metrics-daemon-x7cmt\" (UID: \"25a7353f-76ad-4ac0-92b8-186377723478\") " pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206686    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-dbus\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206706    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-var-lib-cni-bin\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206720    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19a8bde5-584f-4ff9-8e96-df33c5cda82d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206731    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-run-ovn\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:23.206738    5070 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206629    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-run-systemd\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206778    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-var-lib-cni-bin\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:23.206803    5070 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs podName:25a7353f-76ad-4ac0-92b8-186377723478 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:23.706791304 +0000 UTC m=+1.821255242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs") pod "network-metrics-daemon-x7cmt" (UID: "25a7353f-76ad-4ac0-92b8-186377723478") : secret "metrics-daemon-secret" not found
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206754    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-node-log\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206813    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-run-ovn\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206843    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206863    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-system-cni-dir\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206869    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-node-log\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206887    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-multus-cni-dir\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206893    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206914    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-system-cni-dir\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206944    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-sysconfig\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.207003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.206977    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-sysctl-conf\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207026    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-multus-cni-dir\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207035    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-sysconfig\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207062    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67ff6855-c974-451d-b582-679e7f26e98d-host\") pod \"node-ca-w9c2l\" (UID: \"67ff6855-c974-451d-b582-679e7f26e98d\") " pod="openshift-image-registry/node-ca-w9c2l"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207093    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-sysctl-conf\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207097    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-log-socket\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207121    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-run-ovn-kubernetes\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207145    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-registration-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207150    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/67ff6855-c974-451d-b582-679e7f26e98d-host\") pod \"node-ca-w9c2l\" (UID: \"67ff6855-c974-451d-b582-679e7f26e98d\") " pod="openshift-image-registry/node-ca-w9c2l"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207161    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-log-socket\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207171    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-run-ovn-kubernetes\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207173    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207204    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-registration-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207266    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-kubelet\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207286    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-systemd-units\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207302    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-cni-netd\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207325    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-modprobe-d\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207349    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-systemd-units\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207351    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/751205b0-8312-4ea6-920c-9517e19566f2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-grgjj\" (UID: \"751205b0-8312-4ea6-920c-9517e19566f2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grgjj"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207370    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-kubelet\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207393    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-cni-netd\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207395    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-etc-openvswitch\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207417    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-etc-openvswitch\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207438    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207458    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-modprobe-d\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207464    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-cnibin\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207491    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-multus-socket-dir-parent\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207515    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-kubelet-config\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:23.207525    5070 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207529    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-cnibin\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:23.207538    5070 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-67ffc88546-z24dr: secret "image-registry-tls" not found
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207569    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-slash\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207569    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6731c200-9eda-44f2-aa13-2b4a8fbda85e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207593    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df-kubelet-config\") pod \"global-pull-secret-syncer-ljbh2\" (UID: \"dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df\") " pod="kube-system/global-pull-secret-syncer-ljbh2"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207574    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-multus-socket-dir-parent\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:23.207581    5070 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls podName:119141fd-3c4f-4b94-9f18-6892986867e9 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:23.707567202 +0000 UTC m=+1.822031141 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls") pod "image-registry-67ffc88546-z24dr" (UID: "119141fd-3c4f-4b94-9f18-6892986867e9") : secret "image-registry-tls" not found
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207536    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-slash\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207663    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-run-netns\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207680    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-var-lib-kubelet\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207701    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-var-lib-cni-multus\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207718    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-hostroot\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207776    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-hostroot\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207778    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-var-lib-cni-multus\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207790    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-var-lib-kubelet\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207810    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-run-multus-certs\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207807    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-run-netns\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207842    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-socket-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207865    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-kubernetes\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207868    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-run-multus-certs\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207902    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-lib-modules\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207922    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-socket-dir\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207926    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/279b3dd8-58be-43a1-9994-59fa434409f5-metrics-tls\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207951    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-etc-kubernetes\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207918    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-etc-kubernetes\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.207979    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-run-openvswitch\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208001    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-etc-kubernetes\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208003    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-cni-bin\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208038    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-run-openvswitch\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208020    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-lib-modules\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208049    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208073    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19a8bde5-584f-4ff9-8e96-df33c5cda82d-system-cni-dir\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:53:23.208075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208094    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd5c9498-f849-4dd6-a9e7-27a0d5db4fde-host-slash\") pod \"iptables-alerter-kj8jw\" (UID: \"bd5c9498-f849-4dd6-a9e7-27a0d5db4fde\") " pod="openshift-network-operator/iptables-alerter-kj8jw"
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208130    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-host\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:23.208101    5070 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208157    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-multus-conf-dir\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:23.208172    5070 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls podName:6731c200-9eda-44f2-aa13-2b4a8fbda85e nodeName:}" failed. No retries permitted until 2026-11-03 07:53:23.708162166 +0000 UTC m=+1.822626130 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wd497" (UID: "6731c200-9eda-44f2-aa13-2b4a8fbda85e") : secret "insights-runtime-extractor-tls" not found
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208169    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd5c9498-f849-4dd6-a9e7-27a0d5db4fde-host-slash\") pod \"iptables-alerter-kj8jw\" (UID: \"bd5c9498-f849-4dd6-a9e7-27a0d5db4fde\") " pod="openshift-network-operator/iptables-alerter-kj8jw"
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208081    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-cni-bin\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208189    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-host\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208199    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-sys-fs\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208131    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19a8bde5-584f-4ff9-8e96-df33c5cda82d-system-cni-dir\") pod \"multus-additional-cni-plugins-h4dgx\" (UID: \"19a8bde5-584f-4ff9-8e96-df33c5cda82d\") " pod="openshift-multus/multus-additional-cni-plugins-h4dgx"
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208200    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-multus-conf-dir\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208244    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-sys-fs\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208244    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-etc-selinux\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208280    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5738916f-8202-40ed-823e-90f7c9cc4ec9-etc-selinux\") pod \"aws-ebs-csi-driver-node-n28db\" (UID: \"5738916f-8202-40ed-823e-90f7c9cc4ec9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208283    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-run\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208308    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-var-lib-kubelet\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208331    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-run-netns\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208356    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-sys\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208382    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ed628f2-4323-47b6-ac03-b8567812a740-host-var-lib-kubelet\") pod \"multus-msq9p\" (UID: \"2ed628f2-4323-47b6-ac03-b8567812a740\") " pod="openshift-multus/multus-msq9p"
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208392    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/40259634-6b57-492e-8376-75ecf7900b06-host-run-netns\") pod \"ovnkube-node-x6d47\" (UID: \"40259634-6b57-492e-8376-75ecf7900b06\") " pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208386    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-run\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.208406    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/79888519-e73a-4536-96e4-ce8e3ec99037-sys\") pod \"tuned-fgwf4\" (UID: \"79888519-e73a-4536-96e4-ce8e3ec99037\") " pod="openshift-cluster-node-tuning-operator/tuned-fgwf4"
Nov 03 07:53:23.209867 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.209586    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/751205b0-8312-4ea6-920c-9517e19566f2-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-grgjj\" (UID: \"751205b0-8312-4ea6-920c-9517e19566f2\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-grgjj"
Nov 03 07:53:23.210483 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.209934    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/279b3dd8-58be-43a1-9994-59fa434409f5-metrics-tls\") pod \"dns-default-46l5d\" (UID: \"279b3dd8-58be-43a1-9994-59fa434409f5\") " pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:23.280282 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.280233    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"
Nov 03 07:53:23.281628 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.281611    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"
Nov 03 07:53:23.282612 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.282588    5070 scope.go:122] "RemoveContainer" containerID="b17a031689eca024fee8f71239c507cd7dc10ca5e0147ff1e9b4d8440b1d8575"
Nov 03 07:53:23.284616 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.284600    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-grgjj"
Nov 03 07:53:23.302517 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.302499    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:23.577169 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.577117    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2"]
Nov 03 07:53:23.612412 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.612388    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:23.612498 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.612443    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:23.612622 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:23.612606    5070 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle podName:ce890601-127d-4995-9487-635e7aab04f8 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:24.612590524 +0000 UTC m=+2.727054432 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle") pod "router-default-79698c9d55-m2lbh" (UID: "ce890601-127d-4995-9487-635e7aab04f8") : configmap references non-existent config key: service-ca.crt
Nov 03 07:53:23.614445 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.614427    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls\") pod \"image-registry-74774885c5-97jxz\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") " pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:23.624005 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.623985    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks"]
Nov 03 07:53:23.627170 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:23.627149    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc383d3eb_10c7_4fe5_99b6_04c64472ef0a.slice/crio-a170d948b8e9407de1bcb6147dd767dd7aae5ca49200682dd85ca4052fdf93b7 WatchSource:0}: Error finding container a170d948b8e9407de1bcb6147dd767dd7aae5ca49200682dd85ca4052fdf93b7: Status 404 returned error can't find the container with id a170d948b8e9407de1bcb6147dd767dd7aae5ca49200682dd85ca4052fdf93b7
Nov 03 07:53:23.629111 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.629093    5070 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 07:53:23.656025 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.656003    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-46l5d"]
Nov 03 07:53:23.659079 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:23.659059    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279b3dd8_58be_43a1_9994_59fa434409f5.slice/crio-f10e0d9a2ee35439efe0ddb4edf84230a461bca0aaf1d621c5cff5c6d8189153 WatchSource:0}: Error finding container f10e0d9a2ee35439efe0ddb4edf84230a461bca0aaf1d621c5cff5c6d8189153: Status 404 returned error can't find the container with id f10e0d9a2ee35439efe0ddb4edf84230a461bca0aaf1d621c5cff5c6d8189153
Nov 03 07:53:23.684925 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.684898    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-grgjj"]
Nov 03 07:53:23.691335 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:23.691314    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751205b0_8312_4ea6_920c_9517e19566f2.slice/crio-36831cac46512c21e20788cb401d77bbf77b81d8251a97824846a0b4050f71d8 WatchSource:0}: Error finding container 36831cac46512c21e20788cb401d77bbf77b81d8251a97824846a0b4050f71d8: Status 404 returned error can't find the container with id 36831cac46512c21e20788cb401d77bbf77b81d8251a97824846a0b4050f71d8
Nov 03 07:53:23.713355 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.713336    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs\") pod \"network-metrics-daemon-x7cmt\" (UID: \"25a7353f-76ad-4ac0-92b8-186377723478\") " pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:53:23.713436 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.713376    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:23.713436 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.713408    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:23.713505 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:23.713475    5070 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 07:53:23.713535 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:23.713512    5070 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 07:53:23.713564 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:23.713538    5070 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs podName:25a7353f-76ad-4ac0-92b8-186377723478 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:24.713520345 +0000 UTC m=+2.827984267 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs") pod "network-metrics-daemon-x7cmt" (UID: "25a7353f-76ad-4ac0-92b8-186377723478") : secret "metrics-daemon-secret" not found
Nov 03 07:53:23.713564 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:23.713555    5070 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls podName:6731c200-9eda-44f2-aa13-2b4a8fbda85e nodeName:}" failed. No retries permitted until 2026-11-03 07:53:24.713545036 +0000 UTC m=+2.828008944 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wd497" (UID: "6731c200-9eda-44f2-aa13-2b4a8fbda85e") : secret "insights-runtime-extractor-tls" not found
Nov 03 07:53:23.715512 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.715489    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls\") pod \"image-registry-67ffc88546-z24dr\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") " pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:23.884501 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.884475    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:23.902000 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:23.901978    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:24.089690 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:24.089665    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/2.log"
Nov 03 07:53:24.090155 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:24.090141    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/1.log"
Nov 03 07:53:24.090201 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:24.090173    5070 generic.go:356] "Generic (PLEG): container finished" podID="d83648de-81dd-4160-8fa9-06d36b38d9de" containerID="a10b2a1a54d542458744e9c86460bf61c3c55fcc11db69ea74a2a6b9b035fd86" exitCode=255
Nov 03 07:53:24.090275 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:24.090248    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" event={"ID":"d83648de-81dd-4160-8fa9-06d36b38d9de","Type":"ContainerDied","Data":"a10b2a1a54d542458744e9c86460bf61c3c55fcc11db69ea74a2a6b9b035fd86"}
Nov 03 07:53:24.090329 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:24.090316    5070 scope.go:122] "RemoveContainer" containerID="b17a031689eca024fee8f71239c507cd7dc10ca5e0147ff1e9b4d8440b1d8575"
Nov 03 07:53:24.090637 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:24.090615    5070 scope.go:122] "RemoveContainer" containerID="a10b2a1a54d542458744e9c86460bf61c3c55fcc11db69ea74a2a6b9b035fd86"
Nov 03 07:53:24.090880 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:24.090859    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5qpnb_openshift-console-operator(d83648de-81dd-4160-8fa9-06d36b38d9de)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" podUID="d83648de-81dd-4160-8fa9-06d36b38d9de"
Nov 03 07:53:24.091624 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:24.091541    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks" event={"ID":"c383d3eb-10c7-4fe5-99b6-04c64472ef0a","Type":"ContainerStarted","Data":"a170d948b8e9407de1bcb6147dd767dd7aae5ca49200682dd85ca4052fdf93b7"}
Nov 03 07:53:24.092735 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:24.092704    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-grgjj" event={"ID":"751205b0-8312-4ea6-920c-9517e19566f2","Type":"ContainerStarted","Data":"36831cac46512c21e20788cb401d77bbf77b81d8251a97824846a0b4050f71d8"}
Nov 03 07:53:24.093880 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:24.093855    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-46l5d" event={"ID":"279b3dd8-58be-43a1-9994-59fa434409f5","Type":"ContainerStarted","Data":"f10e0d9a2ee35439efe0ddb4edf84230a461bca0aaf1d621c5cff5c6d8189153"}
Nov 03 07:53:24.095035 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:24.095014    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2" event={"ID":"dc162541-2ab5-4475-8856-850a7c5fd437","Type":"ContainerStarted","Data":"4fd998806fc3fb0e0a1f854bf64258e6fd4b8c06f3c0675430a715eefb6c93e8"}
Nov 03 07:53:24.154447 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:24.154395    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74774885c5-97jxz"]
Nov 03 07:53:24.158257 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:24.157671    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf604a27_2a06_42a0_bcbd_f7a10c1420c1.slice/crio-9f593dedc09571c3546d8760b1f3bddea3be656bde40c0b09e367494b2c33392 WatchSource:0}: Error finding container 9f593dedc09571c3546d8760b1f3bddea3be656bde40c0b09e367494b2c33392: Status 404 returned error can't find the container with id 9f593dedc09571c3546d8760b1f3bddea3be656bde40c0b09e367494b2c33392
Nov 03 07:53:24.183030 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:24.183010    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67ffc88546-z24dr"]
Nov 03 07:53:24.393780 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:24.387206    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w9c2l_67ff6855-c974-451d-b582-679e7f26e98d/node-ca/0.log"
Nov 03 07:53:24.621190 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:24.621148    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:24.621486 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:24.621338    5070 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle podName:ce890601-127d-4995-9487-635e7aab04f8 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:26.621315666 +0000 UTC m=+4.735779589 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle") pod "router-default-79698c9d55-m2lbh" (UID: "ce890601-127d-4995-9487-635e7aab04f8") : configmap references non-existent config key: service-ca.crt
Nov 03 07:53:24.722280 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:24.722238    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:24.722453 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:24.722334    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs\") pod \"network-metrics-daemon-x7cmt\" (UID: \"25a7353f-76ad-4ac0-92b8-186377723478\") " pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:53:24.723207 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:24.722861    5070 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 07:53:24.723207 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:24.722928    5070 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls podName:6731c200-9eda-44f2-aa13-2b4a8fbda85e nodeName:}" failed. No retries permitted until 2026-11-03 07:53:26.722913558 +0000 UTC m=+4.837377467 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wd497" (UID: "6731c200-9eda-44f2-aa13-2b4a8fbda85e") : secret "insights-runtime-extractor-tls" not found
Nov 03 07:53:24.726151 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:24.726109    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25a7353f-76ad-4ac0-92b8-186377723478-metrics-certs\") pod \"network-metrics-daemon-x7cmt\" (UID: \"25a7353f-76ad-4ac0-92b8-186377723478\") " pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:53:24.786235 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:24.786198    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7cmt"
Nov 03 07:53:24.788567 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:24.788421    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6ldkm_78b91dbb-d9e9-4524-8827-d74c8c0625d0/serve-healthcheck-canary/0.log"
Nov 03 07:53:24.987929 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:24.987808    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-d7zfg_fc09b315-54d3-44f0-b6ea-8028bd3b212b/migrator/0.log"
Nov 03 07:53:25.074872 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:25.074838    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x7cmt"]
Nov 03 07:53:25.084172 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:25.084135    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a7353f_76ad_4ac0_92b8_186377723478.slice/crio-33f39c69bf0879691b023cc5857002fe65c0ae7e64ffd205e933d910b1608773 WatchSource:0}: Error finding container 33f39c69bf0879691b023cc5857002fe65c0ae7e64ffd205e933d910b1608773: Status 404 returned error can't find the container with id 33f39c69bf0879691b023cc5857002fe65c0ae7e64ffd205e933d910b1608773
Nov 03 07:53:25.105253 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:25.104512    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67ffc88546-z24dr" event={"ID":"119141fd-3c4f-4b94-9f18-6892986867e9","Type":"ContainerStarted","Data":"aaf08aaf001f1fdc49cba51897c692024c67ea83d9a269af54616433a10c309b"}
Nov 03 07:53:25.105253 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:25.104568    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67ffc88546-z24dr" event={"ID":"119141fd-3c4f-4b94-9f18-6892986867e9","Type":"ContainerStarted","Data":"4c629640eedaa3dd010c0fa7bc1ea73eb5773845c207a0f246ad5b208ffcd924"}
Nov 03 07:53:25.107274 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:25.107240    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/2.log"
Nov 03 07:53:25.110022 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:25.109985    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74774885c5-97jxz" event={"ID":"df604a27-2a06-42a0-bcbd-f7a10c1420c1","Type":"ContainerStarted","Data":"a3024eb7124778687f5cfaace0f7b5eebe83f31768c33bc3b423769497a90de9"}
Nov 03 07:53:25.110126 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:25.110036    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74774885c5-97jxz" event={"ID":"df604a27-2a06-42a0-bcbd-f7a10c1420c1","Type":"ContainerStarted","Data":"9f593dedc09571c3546d8760b1f3bddea3be656bde40c0b09e367494b2c33392"}
Nov 03 07:53:25.111735 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:25.111709    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x7cmt" event={"ID":"25a7353f-76ad-4ac0-92b8-186377723478","Type":"ContainerStarted","Data":"33f39c69bf0879691b023cc5857002fe65c0ae7e64ffd205e933d910b1608773"}
Nov 03 07:53:25.186826 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:25.186728    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-fhz8r_1778099a-ebc0-47ba-98d0-2b0e5726e7d2/migrator/0.log"
Nov 03 07:53:25.392669 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:25.392640    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk_e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3/kube-storage-version-migrator-operator/0.log"
Nov 03 07:53:25.672607 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:25.671569    5070 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:25.672607 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:25.672085    5070 scope.go:122] "RemoveContainer" containerID="a10b2a1a54d542458744e9c86460bf61c3c55fcc11db69ea74a2a6b9b035fd86"
Nov 03 07:53:25.672607 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:25.672327    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5qpnb_openshift-console-operator(d83648de-81dd-4160-8fa9-06d36b38d9de)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" podUID="d83648de-81dd-4160-8fa9-06d36b38d9de"
Nov 03 07:53:26.119271 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:26.119234    5070 generic.go:356] "Generic (PLEG): container finished" podID="c620e360-6548-4b0c-ba9a-a9150cf9939c" containerID="c9d39725eee87abc02081ab62a587cea0577e4237fa7bad287fbcc259af7aef1" exitCode=0
Nov 03 07:53:26.119785 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:26.119349    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-crn8p" event={"ID":"c620e360-6548-4b0c-ba9a-a9150cf9939c","Type":"ContainerDied","Data":"c9d39725eee87abc02081ab62a587cea0577e4237fa7bad287fbcc259af7aef1"}
Nov 03 07:53:26.119785 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:26.119682    5070 scope.go:122] "RemoveContainer" containerID="c9d39725eee87abc02081ab62a587cea0577e4237fa7bad287fbcc259af7aef1"
Nov 03 07:53:26.122437 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:26.122418    5070 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 07:53:26.122907 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:26.122812    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-grgjj" event={"ID":"751205b0-8312-4ea6-920c-9517e19566f2","Type":"ContainerStarted","Data":"76f4d5be6bb01b723e4fb187c8c66d5c69978c7fb4ce3347091ef7aaecc8b31a"}
Nov 03 07:53:26.122907 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:26.122841    5070 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 07:53:26.641571 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:26.641524    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:26.641665 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:26.641648    5070 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle podName:ce890601-127d-4995-9487-635e7aab04f8 nodeName:}" failed. No retries permitted until 2026-11-03 07:53:30.641633558 +0000 UTC m=+8.756097469 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle") pod "router-default-79698c9d55-m2lbh" (UID: "ce890601-127d-4995-9487-635e7aab04f8") : configmap references non-existent config key: service-ca.crt
Nov 03 07:53:26.741964 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:26.741929    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:26.742119 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:26.742094    5070 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 07:53:26.742193 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:26.742173    5070 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls podName:6731c200-9eda-44f2-aa13-2b4a8fbda85e nodeName:}" failed. No retries permitted until 2026-11-03 07:53:30.742153131 +0000 UTC m=+8.856617042 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wd497" (UID: "6731c200-9eda-44f2-aa13-2b4a8fbda85e") : secret "insights-runtime-extractor-tls" not found
Nov 03 07:53:27.128037 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:27.127976    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-f68ks" event={"ID":"c383d3eb-10c7-4fe5-99b6-04c64472ef0a","Type":"ContainerStarted","Data":"0638c8337c26749320bab67dae488e445918cc42246f5b78a20914fda6d6f79d"}
Nov 03 07:53:27.129817 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:27.129790    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x7cmt" event={"ID":"25a7353f-76ad-4ac0-92b8-186377723478","Type":"ContainerStarted","Data":"47056537eafb64e06c56340becf8f19b80df1f8d382d77d2d5b0cae36d3c012d"}
Nov 03 07:53:27.129971 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:27.129843    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x7cmt" event={"ID":"25a7353f-76ad-4ac0-92b8-186377723478","Type":"ContainerStarted","Data":"f7f4c723526b492df10fee1f1286bf509dd54c7db7a9f7422b5bcd9b190f0fe3"}
Nov 03 07:53:27.131843 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:27.131819    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-46l5d" event={"ID":"279b3dd8-58be-43a1-9994-59fa434409f5","Type":"ContainerStarted","Data":"262d481320498c94319205e9163d456386738a4e906c3e1bc275f4bb5eb54b46"}
Nov 03 07:53:27.131945 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:27.131857    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-46l5d" event={"ID":"279b3dd8-58be-43a1-9994-59fa434409f5","Type":"ContainerStarted","Data":"99e580f7dfdb7c4307970694f2f3de8a5190c6c2068e78637b28806fdd171bfe"}
Nov 03 07:53:27.131945 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:27.131878    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:27.133552 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:27.133525    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2" event={"ID":"dc162541-2ab5-4475-8856-850a7c5fd437","Type":"ContainerStarted","Data":"0154e17bc990c75820e3a1242b008717f05f924f3b3bff16bd083c258ed16a5a"}
Nov 03 07:53:27.133670 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:27.133571    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-dtgx2" event={"ID":"dc162541-2ab5-4475-8856-850a7c5fd437","Type":"ContainerStarted","Data":"4bfc53fa661e264fa20d4c007716125754200d87102740af68339c17fd183448"}
Nov 03 07:53:27.135229 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:27.135204    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-crn8p" event={"ID":"c620e360-6548-4b0c-ba9a-a9150cf9939c","Type":"ContainerStarted","Data":"e542387459356bd587cd6f2ec5cef8f56c491dfdd3570262fe4e337399b9481c"}
Nov 03 07:53:28.867552 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:28.867521    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:29.012584 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:29.012549    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:29.012949 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:29.012931    5070 scope.go:122] "RemoveContainer" containerID="a10b2a1a54d542458744e9c86460bf61c3c55fcc11db69ea74a2a6b9b035fd86"
Nov 03 07:53:29.013128 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:29.013107    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-5qpnb_openshift-console-operator(d83648de-81dd-4160-8fa9-06d36b38d9de)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" podUID="d83648de-81dd-4160-8fa9-06d36b38d9de"
Nov 03 07:53:30.671317 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:30.671279    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:30.671921 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:30.671899    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce890601-127d-4995-9487-635e7aab04f8-service-ca-bundle\") pod \"router-default-79698c9d55-m2lbh\" (UID: \"ce890601-127d-4995-9487-635e7aab04f8\") " pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:30.772550 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:30.772518    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:30.774897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:30.774868    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6731c200-9eda-44f2-aa13-2b4a8fbda85e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wd497\" (UID: \"6731c200-9eda-44f2-aa13-2b4a8fbda85e\") " pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:30.780832 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:30.780808    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:30.803350 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:30.803328    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wd497"
Nov 03 07:53:30.856550 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:30.856209    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:53:31.058112 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:31.058089    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-79698c9d55-m2lbh"]
Nov 03 07:53:31.059454 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:31.059422    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce890601_127d_4995_9487_635e7aab04f8.slice/crio-756023c5279947bd68b96eba37af4fdf0f2fffc725eb9cee5bfbd2296ded3f00 WatchSource:0}: Error finding container 756023c5279947bd68b96eba37af4fdf0f2fffc725eb9cee5bfbd2296ded3f00: Status 404 returned error can't find the container with id 756023c5279947bd68b96eba37af4fdf0f2fffc725eb9cee5bfbd2296ded3f00
Nov 03 07:53:31.077601 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:31.077569    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wd497"]
Nov 03 07:53:31.085847 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:31.085822    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6731c200_9eda_44f2_aa13_2b4a8fbda85e.slice/crio-5ce2bba6db05bfca39048b77ff83f565381731a72eaa364f2a757e61f87ff1c7 WatchSource:0}: Error finding container 5ce2bba6db05bfca39048b77ff83f565381731a72eaa364f2a757e61f87ff1c7: Status 404 returned error can't find the container with id 5ce2bba6db05bfca39048b77ff83f565381731a72eaa364f2a757e61f87ff1c7
Nov 03 07:53:31.155830 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:31.155753    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wd497" event={"ID":"6731c200-9eda-44f2-aa13-2b4a8fbda85e","Type":"ContainerStarted","Data":"226fab497528107b7f18c8ba74a76098c3bb8c3515a60dd716c03b1f45edc3a0"}
Nov 03 07:53:31.156053 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:31.156002    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wd497" event={"ID":"6731c200-9eda-44f2-aa13-2b4a8fbda85e","Type":"ContainerStarted","Data":"5ce2bba6db05bfca39048b77ff83f565381731a72eaa364f2a757e61f87ff1c7"}
Nov 03 07:53:31.158083 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:31.157467    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79698c9d55-m2lbh" event={"ID":"ce890601-127d-4995-9487-635e7aab04f8","Type":"ContainerStarted","Data":"b716061986c42f36c0ded6627d38340288fc785f32b3ea9da22c3b67a81d7727"}
Nov 03 07:53:31.158083 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:31.157502    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-79698c9d55-m2lbh" event={"ID":"ce890601-127d-4995-9487-635e7aab04f8","Type":"ContainerStarted","Data":"756023c5279947bd68b96eba37af4fdf0f2fffc725eb9cee5bfbd2296ded3f00"}
Nov 03 07:53:31.541588 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:31.541558    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:31.781849 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:31.781815    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:31.784679 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:31.784654    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:32.162828 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:32.162794    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wd497" event={"ID":"6731c200-9eda-44f2-aa13-2b4a8fbda85e","Type":"ContainerStarted","Data":"d4c2dfcc4b88c18660956404d10a27acf6050849029a06f6441345426909a65a"}
Nov 03 07:53:32.444118 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:32.444049    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:32.969827 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:32.969798    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:32.970164 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:32.969998    5070 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 07:53:33.166726 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:33.166691    5070 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 07:53:33.201815 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:33.201789    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x6d47"
Nov 03 07:53:35.018239 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:35.018207    5070 scope.go:122] "RemoveContainer" containerID="a10b2a1a54d542458744e9c86460bf61c3c55fcc11db69ea74a2a6b9b035fd86"
Nov 03 07:53:35.176625 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:35.176594    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wd497" event={"ID":"6731c200-9eda-44f2-aa13-2b4a8fbda85e","Type":"ContainerStarted","Data":"6ec08f0ab99c7265fc7fadd11ab4660848368dadd1daa38d28bb6bf91dd7301c"}
Nov 03 07:53:35.178423 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:35.178404    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/2.log"
Nov 03 07:53:35.178530 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:35.178456    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" event={"ID":"d83648de-81dd-4160-8fa9-06d36b38d9de","Type":"ContainerStarted","Data":"41db54c2e9197838faec3ccd184adac0bf784c452fd80ef24684e43bbab81f7a"}
Nov 03 07:53:35.178724 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:35.178701    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:35.274138 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:35.274083    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 07:53:36.935490 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:36.935450    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-kfr2h-k95nf"]
Nov 03 07:53:36.944053 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:36.944026    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kfr2h-k95nf"
Nov 03 07:53:36.957984 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:36.957956    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-kfr2h-k95nf"]
Nov 03 07:53:36.967543 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:36.967521    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74774885c5-97jxz"]
Nov 03 07:53:36.967744 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:36.967730    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-qmpgn"]
Nov 03 07:53:36.973190 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:36.973171    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j"]
Nov 03 07:53:36.973335 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:36.973232    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-qmpgn"
Nov 03 07:53:36.979800 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:36.979781    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j"
Nov 03 07:53:36.986476 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:36.983605    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-6hd76\""
Nov 03 07:53:36.986476 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:36.983903    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 07:53:36.986476 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:36.983968    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 07:53:36.986476 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:36.985647    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j"]
Nov 03 07:53:36.986476 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:36.986430    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-qmpgn"]
Nov 03 07:53:37.030735 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.030708    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psrfk\" (UniqueName: \"kubernetes.io/projected/5400462b-2298-41e2-b44b-caa2f3a1e496-kube-api-access-psrfk\") pod \"downloads-6dd789c8b7-qmpgn\" (UID: \"5400462b-2298-41e2-b44b-caa2f3a1e496\") " pod="openshift-console/downloads-6dd789c8b7-qmpgn"
Nov 03 07:53:37.030891 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.030749    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks4mg\" (UniqueName: \"kubernetes.io/projected/2c197040-3bfe-416c-be23-5d5b67678173-kube-api-access-ks4mg\") pod \"periodic-gathering-kfr2h-k95nf\" (UID: \"2c197040-3bfe-416c-be23-5d5b67678173\") " pod="openshift-insights/periodic-gathering-kfr2h-k95nf"
Nov 03 07:53:37.030891 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.030870    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5400462b-2298-41e2-b44b-caa2f3a1e496-tmp\") pod \"downloads-6dd789c8b7-qmpgn\" (UID: \"5400462b-2298-41e2-b44b-caa2f3a1e496\") " pod="openshift-console/downloads-6dd789c8b7-qmpgn"
Nov 03 07:53:37.031018 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.030996    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/22591a72-cf89-4406-8e20-82fd6d0f6e6a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-j5h2j\" (UID: \"22591a72-cf89-4406-8e20-82fd6d0f6e6a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j"
Nov 03 07:53:37.031057 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.031045    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c197040-3bfe-416c-be23-5d5b67678173-service-ca-bundle\") pod \"periodic-gathering-kfr2h-k95nf\" (UID: \"2c197040-3bfe-416c-be23-5d5b67678173\") " pod="openshift-insights/periodic-gathering-kfr2h-k95nf"
Nov 03 07:53:37.031096 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.031084    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c197040-3bfe-416c-be23-5d5b67678173-trusted-ca-bundle\") pod \"periodic-gathering-kfr2h-k95nf\" (UID: \"2c197040-3bfe-416c-be23-5d5b67678173\") " pod="openshift-insights/periodic-gathering-kfr2h-k95nf"
Nov 03 07:53:37.031130 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.031113    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2c197040-3bfe-416c-be23-5d5b67678173-archives-path\") pod \"periodic-gathering-kfr2h-k95nf\" (UID: \"2c197040-3bfe-416c-be23-5d5b67678173\") " pod="openshift-insights/periodic-gathering-kfr2h-k95nf"
Nov 03 07:53:37.076927 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.076897    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7d9549b467-hqhvz"]
Nov 03 07:53:37.080389 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.080370    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.081677 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.081648    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2hhpp\""
Nov 03 07:53:37.081922 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.081893    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 03 07:53:37.096776 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.096693    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d9549b467-hqhvz"]
Nov 03 07:53:37.131528 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.131504    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c197040-3bfe-416c-be23-5d5b67678173-trusted-ca-bundle\") pod \"periodic-gathering-kfr2h-k95nf\" (UID: \"2c197040-3bfe-416c-be23-5d5b67678173\") " pod="openshift-insights/periodic-gathering-kfr2h-k95nf"
Nov 03 07:53:37.131671 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.131535    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2c197040-3bfe-416c-be23-5d5b67678173-archives-path\") pod \"periodic-gathering-kfr2h-k95nf\" (UID: \"2c197040-3bfe-416c-be23-5d5b67678173\") " pod="openshift-insights/periodic-gathering-kfr2h-k95nf"
Nov 03 07:53:37.131671 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.131559    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5211e023-fa82-4010-939e-89afebb9e6b8-installation-pull-secrets\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.131671 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.131607    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psrfk\" (UniqueName: \"kubernetes.io/projected/5400462b-2298-41e2-b44b-caa2f3a1e496-kube-api-access-psrfk\") pod \"downloads-6dd789c8b7-qmpgn\" (UID: \"5400462b-2298-41e2-b44b-caa2f3a1e496\") " pod="openshift-console/downloads-6dd789c8b7-qmpgn"
Nov 03 07:53:37.131671 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.131652    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5211e023-fa82-4010-939e-89afebb9e6b8-ca-trust-extracted\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.132040 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.131919    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5211e023-fa82-4010-939e-89afebb9e6b8-trusted-ca\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.132040 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.131968    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75zpg\" (UniqueName: \"kubernetes.io/projected/5211e023-fa82-4010-939e-89afebb9e6b8-kube-api-access-75zpg\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.132115 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.132076    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2c197040-3bfe-416c-be23-5d5b67678173-archives-path\") pod \"periodic-gathering-kfr2h-k95nf\" (UID: \"2c197040-3bfe-416c-be23-5d5b67678173\") " pod="openshift-insights/periodic-gathering-kfr2h-k95nf"
Nov 03 07:53:37.132115 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.132093    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks4mg\" (UniqueName: \"kubernetes.io/projected/2c197040-3bfe-416c-be23-5d5b67678173-kube-api-access-ks4mg\") pod \"periodic-gathering-kfr2h-k95nf\" (UID: \"2c197040-3bfe-416c-be23-5d5b67678173\") " pod="openshift-insights/periodic-gathering-kfr2h-k95nf"
Nov 03 07:53:37.132185 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.132148    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5211e023-fa82-4010-939e-89afebb9e6b8-bound-sa-token\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.132229 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.132183    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5400462b-2298-41e2-b44b-caa2f3a1e496-tmp\") pod \"downloads-6dd789c8b7-qmpgn\" (UID: \"5400462b-2298-41e2-b44b-caa2f3a1e496\") " pod="openshift-console/downloads-6dd789c8b7-qmpgn"
Nov 03 07:53:37.132229 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.132209    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5211e023-fa82-4010-939e-89afebb9e6b8-registry-tls\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.132304 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.132264    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/22591a72-cf89-4406-8e20-82fd6d0f6e6a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-j5h2j\" (UID: \"22591a72-cf89-4406-8e20-82fd6d0f6e6a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j"
Nov 03 07:53:37.132304 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.132294    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5211e023-fa82-4010-939e-89afebb9e6b8-image-registry-private-configuration\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.132398 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.132333    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c197040-3bfe-416c-be23-5d5b67678173-service-ca-bundle\") pod \"periodic-gathering-kfr2h-k95nf\" (UID: \"2c197040-3bfe-416c-be23-5d5b67678173\") " pod="openshift-insights/periodic-gathering-kfr2h-k95nf"
Nov 03 07:53:37.132510 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.132487    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5211e023-fa82-4010-939e-89afebb9e6b8-registry-certificates\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.132579 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.132560    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c197040-3bfe-416c-be23-5d5b67678173-trusted-ca-bundle\") pod \"periodic-gathering-kfr2h-k95nf\" (UID: \"2c197040-3bfe-416c-be23-5d5b67678173\") " pod="openshift-insights/periodic-gathering-kfr2h-k95nf"
Nov 03 07:53:37.132666 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.132649    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5400462b-2298-41e2-b44b-caa2f3a1e496-tmp\") pod \"downloads-6dd789c8b7-qmpgn\" (UID: \"5400462b-2298-41e2-b44b-caa2f3a1e496\") " pod="openshift-console/downloads-6dd789c8b7-qmpgn"
Nov 03 07:53:37.133070 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.133051    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c197040-3bfe-416c-be23-5d5b67678173-service-ca-bundle\") pod \"periodic-gathering-kfr2h-k95nf\" (UID: \"2c197040-3bfe-416c-be23-5d5b67678173\") " pod="openshift-insights/periodic-gathering-kfr2h-k95nf"
Nov 03 07:53:37.136414 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.136388    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/22591a72-cf89-4406-8e20-82fd6d0f6e6a-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-j5h2j\" (UID: \"22591a72-cf89-4406-8e20-82fd6d0f6e6a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j"
Nov 03 07:53:37.140744 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.140724    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psrfk\" (UniqueName: \"kubernetes.io/projected/5400462b-2298-41e2-b44b-caa2f3a1e496-kube-api-access-psrfk\") pod \"downloads-6dd789c8b7-qmpgn\" (UID: \"5400462b-2298-41e2-b44b-caa2f3a1e496\") " pod="openshift-console/downloads-6dd789c8b7-qmpgn"
Nov 03 07:53:37.141561 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.140960    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks4mg\" (UniqueName: \"kubernetes.io/projected/2c197040-3bfe-416c-be23-5d5b67678173-kube-api-access-ks4mg\") pod \"periodic-gathering-kfr2h-k95nf\" (UID: \"2c197040-3bfe-416c-be23-5d5b67678173\") " pod="openshift-insights/periodic-gathering-kfr2h-k95nf"
Nov 03 07:53:37.142362 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.142342    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-46l5d"
Nov 03 07:53:37.233211 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.233126    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5211e023-fa82-4010-939e-89afebb9e6b8-image-registry-private-configuration\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.233211 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.233174    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5211e023-fa82-4010-939e-89afebb9e6b8-registry-certificates\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.233434 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.233213    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5211e023-fa82-4010-939e-89afebb9e6b8-installation-pull-secrets\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.233434 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.233320    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5211e023-fa82-4010-939e-89afebb9e6b8-ca-trust-extracted\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.233434 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.233347    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5211e023-fa82-4010-939e-89afebb9e6b8-trusted-ca\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.233434 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.233375    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75zpg\" (UniqueName: \"kubernetes.io/projected/5211e023-fa82-4010-939e-89afebb9e6b8-kube-api-access-75zpg\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.233434 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.233409    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5211e023-fa82-4010-939e-89afebb9e6b8-bound-sa-token\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.233685 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.233457    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5211e023-fa82-4010-939e-89afebb9e6b8-registry-tls\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.234006 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.233986    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5211e023-fa82-4010-939e-89afebb9e6b8-ca-trust-extracted\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.234160 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.234052    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5211e023-fa82-4010-939e-89afebb9e6b8-registry-certificates\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.234745 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.234724    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5211e023-fa82-4010-939e-89afebb9e6b8-trusted-ca\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.236189 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.236162    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5211e023-fa82-4010-939e-89afebb9e6b8-installation-pull-secrets\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.236407 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.236381    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5211e023-fa82-4010-939e-89afebb9e6b8-registry-tls\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.236963 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.236944    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5211e023-fa82-4010-939e-89afebb9e6b8-image-registry-private-configuration\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.243288 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.243257    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5211e023-fa82-4010-939e-89afebb9e6b8-bound-sa-token\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.243856 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.243837    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75zpg\" (UniqueName: \"kubernetes.io/projected/5211e023-fa82-4010-939e-89afebb9e6b8-kube-api-access-75zpg\") pod \"image-registry-7d9549b467-hqhvz\" (UID: \"5211e023-fa82-4010-939e-89afebb9e6b8\") " pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.259281 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.259255    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kfr2h-k95nf"
Nov 03 07:53:37.297815 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.297792    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-qmpgn"
Nov 03 07:53:37.301099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.301077    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j"
Nov 03 07:53:37.397861 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.396215    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:37.563015 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.562985    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-kfr2h-k95nf"]
Nov 03 07:53:37.566097 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:37.566067    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c197040_3bfe_416c_be23_5d5b67678173.slice/crio-02a0585ede1251e50d2abdba9af0b2fba7aaa691d5836c9cd0779d87bb79f69b WatchSource:0}: Error finding container 02a0585ede1251e50d2abdba9af0b2fba7aaa691d5836c9cd0779d87bb79f69b: Status 404 returned error can't find the container with id 02a0585ede1251e50d2abdba9af0b2fba7aaa691d5836c9cd0779d87bb79f69b
Nov 03 07:53:37.596152 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.596131    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-qmpgn"]
Nov 03 07:53:37.600335 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:37.600313    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5400462b_2298_41e2_b44b_caa2f3a1e496.slice/crio-599a6cb67a03e78cbfec88b90b13974cae9fc94850203951c3ae09df60d23ca7 WatchSource:0}: Error finding container 599a6cb67a03e78cbfec88b90b13974cae9fc94850203951c3ae09df60d23ca7: Status 404 returned error can't find the container with id 599a6cb67a03e78cbfec88b90b13974cae9fc94850203951c3ae09df60d23ca7
Nov 03 07:53:37.631012 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.630989    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j"]
Nov 03 07:53:37.634048 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:37.634026    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22591a72_cf89_4406_8e20_82fd6d0f6e6a.slice/crio-d743865b8f8216f0b974ecd70de6957d4baad8f066fc58e9e8e189c7ab3ca9e3 WatchSource:0}: Error finding container d743865b8f8216f0b974ecd70de6957d4baad8f066fc58e9e8e189c7ab3ca9e3: Status 404 returned error can't find the container with id d743865b8f8216f0b974ecd70de6957d4baad8f066fc58e9e8e189c7ab3ca9e3
Nov 03 07:53:37.708452 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:37.708382    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7d9549b467-hqhvz"]
Nov 03 07:53:37.712594 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:37.712568    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5211e023_fa82_4010_939e_89afebb9e6b8.slice/crio-aba143bf6b1dd684f1d75137b16ffa10ab741da1d500d86ea33832134ed95b7d WatchSource:0}: Error finding container aba143bf6b1dd684f1d75137b16ffa10ab741da1d500d86ea33832134ed95b7d: Status 404 returned error can't find the container with id aba143bf6b1dd684f1d75137b16ffa10ab741da1d500d86ea33832134ed95b7d
Nov 03 07:53:38.206649 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:38.206609    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d9549b467-hqhvz" event={"ID":"5211e023-fa82-4010-939e-89afebb9e6b8","Type":"ContainerStarted","Data":"a0f5d44418a8ed4fdedf0b5f3452626063bda534bdd4c631195fca4a944673aa"}
Nov 03 07:53:38.207125 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:38.206671    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d9549b467-hqhvz" event={"ID":"5211e023-fa82-4010-939e-89afebb9e6b8","Type":"ContainerStarted","Data":"aba143bf6b1dd684f1d75137b16ffa10ab741da1d500d86ea33832134ed95b7d"}
Nov 03 07:53:38.207125 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:38.206735    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:38.208549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:38.208524    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-qmpgn" event={"ID":"5400462b-2298-41e2-b44b-caa2f3a1e496","Type":"ContainerStarted","Data":"599a6cb67a03e78cbfec88b90b13974cae9fc94850203951c3ae09df60d23ca7"}
Nov 03 07:53:38.210752 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:38.210282    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kfr2h-k95nf" event={"ID":"2c197040-3bfe-416c-be23-5d5b67678173","Type":"ContainerStarted","Data":"9cb92312aadc9a572f614677df0e2b56a829ba9a81fbce2e8230dc65f5292b8e"}
Nov 03 07:53:38.210848 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:38.210801    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kfr2h-k95nf" event={"ID":"2c197040-3bfe-416c-be23-5d5b67678173","Type":"ContainerStarted","Data":"02a0585ede1251e50d2abdba9af0b2fba7aaa691d5836c9cd0779d87bb79f69b"}
Nov 03 07:53:38.212028 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:38.212001    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j" event={"ID":"22591a72-cf89-4406-8e20-82fd6d0f6e6a","Type":"ContainerStarted","Data":"d743865b8f8216f0b974ecd70de6957d4baad8f066fc58e9e8e189c7ab3ca9e3"}
Nov 03 07:53:38.223136 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:38.223082    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7d9549b467-hqhvz" podStartSLOduration=1.223070538 podStartE2EDuration="1.223070538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:38.22303086 +0000 UTC m=+16.337494792" watchObservedRunningTime="2026-11-03 07:53:38.223070538 +0000 UTC m=+16.337534466"
Nov 03 07:53:38.239051 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:38.239004    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-kfr2h-k95nf" podStartSLOduration=2.238990772 podStartE2EDuration="2.238990772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:38.237976644 +0000 UTC m=+16.352440588" watchObservedRunningTime="2026-11-03 07:53:38.238990772 +0000 UTC m=+16.353454704"
Nov 03 07:53:38.386737 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:38.386706    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-46l5d_279b3dd8-58be-43a1-9994-59fa434409f5/dns/0.log"
Nov 03 07:53:38.584190 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:38.584066    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-46l5d_279b3dd8-58be-43a1-9994-59fa434409f5/kube-rbac-proxy/0.log"
Nov 03 07:53:38.783114 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:38.783085    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-drdft_f142ef3a-aa21-48eb-a403-8d51d843c72d/dns-node-resolver/0.log"
Nov 03 07:53:39.218918 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:39.218796    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j" event={"ID":"22591a72-cf89-4406-8e20-82fd6d0f6e6a","Type":"ContainerStarted","Data":"d9fea37b92890625680a72e06e7f19262c344e2ce9b5941ca192b5f45e22ca33"}
Nov 03 07:53:39.219416 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:39.219398    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j"
Nov 03 07:53:39.224395 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:39.224372    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j"
Nov 03 07:53:39.232004 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:39.231740    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j" podStartSLOduration=1.925202265 podStartE2EDuration="3.231726578s" totalImagesPullingTime="1.306524313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:39.231485629 +0000 UTC m=+17.345949561" watchObservedRunningTime="2026-11-03 07:53:39.231726578 +0000 UTC m=+17.346190511"
Nov 03 07:53:39.383954 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:39.383922    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-67ffc88546-z24dr_119141fd-3c4f-4b94-9f18-6892986867e9/registry/0.log"
Nov 03 07:53:39.433806 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:39.433782    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:39.433930 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:39.433902    5070 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 07:53:39.435634 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:39.435614    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 07:53:39.584227 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:39.584155    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-74774885c5-97jxz_df604a27-2a06-42a0-bcbd-f7a10c1420c1/registry/0.log"
Nov 03 07:53:39.784337 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:39.784306    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7d9549b467-hqhvz_5211e023-fa82-4010-939e-89afebb9e6b8/registry/0.log"
Nov 03 07:53:40.174977 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.174946    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-l99g7"]
Nov 03 07:53:40.178799 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.178751    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-l99g7"
Nov 03 07:53:40.181155 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.181115    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 07:53:40.181343 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.181202    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 07:53:40.181564 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.181520    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-mhtlm\""
Nov 03 07:53:40.181666 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.181612    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 07:53:40.186002 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.185977    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-l99g7"]
Nov 03 07:53:40.263572 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.263536    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bcbcf547-bdcb-46f9-bd2c-9164868a80ec-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-l99g7\" (UID: \"bcbcf547-bdcb-46f9-bd2c-9164868a80ec\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l99g7"
Nov 03 07:53:40.263972 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.263583    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcbcf547-bdcb-46f9-bd2c-9164868a80ec-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-l99g7\" (UID: \"bcbcf547-bdcb-46f9-bd2c-9164868a80ec\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l99g7"
Nov 03 07:53:40.263972 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.263694    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr4bx\" (UniqueName: \"kubernetes.io/projected/bcbcf547-bdcb-46f9-bd2c-9164868a80ec-kube-api-access-lr4bx\") pod \"prometheus-operator-9cd89b74c-l99g7\" (UID: \"bcbcf547-bdcb-46f9-bd2c-9164868a80ec\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l99g7"
Nov 03 07:53:40.263972 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.263723    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bcbcf547-bdcb-46f9-bd2c-9164868a80ec-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-l99g7\" (UID: \"bcbcf547-bdcb-46f9-bd2c-9164868a80ec\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l99g7"
Nov 03 07:53:40.364432 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.364401    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bcbcf547-bdcb-46f9-bd2c-9164868a80ec-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-l99g7\" (UID: \"bcbcf547-bdcb-46f9-bd2c-9164868a80ec\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l99g7"
Nov 03 07:53:40.364592 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.364441    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcbcf547-bdcb-46f9-bd2c-9164868a80ec-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-l99g7\" (UID: \"bcbcf547-bdcb-46f9-bd2c-9164868a80ec\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l99g7"
Nov 03 07:53:40.365097 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.364817    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr4bx\" (UniqueName: \"kubernetes.io/projected/bcbcf547-bdcb-46f9-bd2c-9164868a80ec-kube-api-access-lr4bx\") pod \"prometheus-operator-9cd89b74c-l99g7\" (UID: \"bcbcf547-bdcb-46f9-bd2c-9164868a80ec\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l99g7"
Nov 03 07:53:40.365097 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.364862    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bcbcf547-bdcb-46f9-bd2c-9164868a80ec-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-l99g7\" (UID: \"bcbcf547-bdcb-46f9-bd2c-9164868a80ec\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l99g7"
Nov 03 07:53:40.365394 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.365344    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bcbcf547-bdcb-46f9-bd2c-9164868a80ec-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-l99g7\" (UID: \"bcbcf547-bdcb-46f9-bd2c-9164868a80ec\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l99g7"
Nov 03 07:53:40.367250 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.367228    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bcbcf547-bdcb-46f9-bd2c-9164868a80ec-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-l99g7\" (UID: \"bcbcf547-bdcb-46f9-bd2c-9164868a80ec\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l99g7"
Nov 03 07:53:40.367471 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.367447    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/bcbcf547-bdcb-46f9-bd2c-9164868a80ec-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-l99g7\" (UID: \"bcbcf547-bdcb-46f9-bd2c-9164868a80ec\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l99g7"
Nov 03 07:53:40.372312 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.372289    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr4bx\" (UniqueName: \"kubernetes.io/projected/bcbcf547-bdcb-46f9-bd2c-9164868a80ec-kube-api-access-lr4bx\") pod \"prometheus-operator-9cd89b74c-l99g7\" (UID: \"bcbcf547-bdcb-46f9-bd2c-9164868a80ec\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-l99g7"
Nov 03 07:53:40.383294 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.383267    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w9c2l_67ff6855-c974-451d-b582-679e7f26e98d/node-ca/0.log"
Nov 03 07:53:40.512743 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.512689    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-l99g7"
Nov 03 07:53:40.606264 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.606223    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79698c9d55-m2lbh_ce890601-127d-4995-9487-635e7aab04f8/router/0.log"
Nov 03 07:53:40.783780 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.783687    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6ldkm_78b91dbb-d9e9-4524-8827-d74c8c0625d0/serve-healthcheck-canary/0.log"
Nov 03 07:53:40.792639 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:40.792610    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-l99g7"]
Nov 03 07:53:40.797346 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:40.797316    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcbcf547_bdcb_46f9_bd2c_9164868a80ec.slice/crio-840f448e6ffe33a1e0f1ee8666aaf412eddfb51f70a5658ae505abe283be0a3d WatchSource:0}: Error finding container 840f448e6ffe33a1e0f1ee8666aaf412eddfb51f70a5658ae505abe283be0a3d: Status 404 returned error can't find the container with id 840f448e6ffe33a1e0f1ee8666aaf412eddfb51f70a5658ae505abe283be0a3d
Nov 03 07:53:41.231790 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:41.231738    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-l99g7" event={"ID":"bcbcf547-bdcb-46f9-bd2c-9164868a80ec","Type":"ContainerStarted","Data":"840f448e6ffe33a1e0f1ee8666aaf412eddfb51f70a5658ae505abe283be0a3d"}
Nov 03 07:53:41.860223 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:41.860190    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f6df9f67-6dfhh"]
Nov 03 07:53:41.864564 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:41.864117    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:41.866873 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:41.866847    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 07:53:41.868109 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:41.868082    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 07:53:41.868109 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:41.868097    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 07:53:41.868298 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:41.868265    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 07:53:41.868298 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:41.868281    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 07:53:41.868386 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:41.868332    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bj94p\""
Nov 03 07:53:41.872429 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:41.872405    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f6df9f67-6dfhh"]
Nov 03 07:53:41.979104 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:41.979073    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8da3844c-03db-451b-87c0-96f1c62167d9-console-serving-cert\") pod \"console-5f6df9f67-6dfhh\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") " pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:41.979104 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:41.979106    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8da3844c-03db-451b-87c0-96f1c62167d9-service-ca\") pod \"console-5f6df9f67-6dfhh\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") " pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:41.979350 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:41.979123    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8da3844c-03db-451b-87c0-96f1c62167d9-console-oauth-config\") pod \"console-5f6df9f67-6dfhh\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") " pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:41.979350 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:41.979149    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zp9d\" (UniqueName: \"kubernetes.io/projected/8da3844c-03db-451b-87c0-96f1c62167d9-kube-api-access-8zp9d\") pod \"console-5f6df9f67-6dfhh\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") " pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:41.979350 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:41.979274    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8da3844c-03db-451b-87c0-96f1c62167d9-console-config\") pod \"console-5f6df9f67-6dfhh\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") " pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:41.979461 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:41.979366    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8da3844c-03db-451b-87c0-96f1c62167d9-tmp\") pod \"console-5f6df9f67-6dfhh\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") " pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:41.979461 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:41.979398    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8da3844c-03db-451b-87c0-96f1c62167d9-oauth-serving-cert\") pod \"console-5f6df9f67-6dfhh\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") " pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:42.080582 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:42.080540    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zp9d\" (UniqueName: \"kubernetes.io/projected/8da3844c-03db-451b-87c0-96f1c62167d9-kube-api-access-8zp9d\") pod \"console-5f6df9f67-6dfhh\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") " pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:42.080753 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:42.080595    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8da3844c-03db-451b-87c0-96f1c62167d9-console-config\") pod \"console-5f6df9f67-6dfhh\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") " pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:42.080753 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:42.080624    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8da3844c-03db-451b-87c0-96f1c62167d9-tmp\") pod \"console-5f6df9f67-6dfhh\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") " pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:42.080753 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:42.080651    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8da3844c-03db-451b-87c0-96f1c62167d9-oauth-serving-cert\") pod \"console-5f6df9f67-6dfhh\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") " pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:42.080753 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:42.080749    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8da3844c-03db-451b-87c0-96f1c62167d9-console-serving-cert\") pod \"console-5f6df9f67-6dfhh\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") " pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:42.080989 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:42.080788    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8da3844c-03db-451b-87c0-96f1c62167d9-service-ca\") pod \"console-5f6df9f67-6dfhh\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") " pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:42.080989 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:42.080807    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8da3844c-03db-451b-87c0-96f1c62167d9-console-oauth-config\") pod \"console-5f6df9f67-6dfhh\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") " pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:42.081440 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:42.081406    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8da3844c-03db-451b-87c0-96f1c62167d9-tmp\") pod \"console-5f6df9f67-6dfhh\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") " pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:42.081812 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:42.081790    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8da3844c-03db-451b-87c0-96f1c62167d9-oauth-serving-cert\") pod \"console-5f6df9f67-6dfhh\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") " pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:42.081911 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:42.081806    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8da3844c-03db-451b-87c0-96f1c62167d9-console-config\") pod \"console-5f6df9f67-6dfhh\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") " pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:42.081911 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:42.081817    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8da3844c-03db-451b-87c0-96f1c62167d9-service-ca\") pod \"console-5f6df9f67-6dfhh\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") " pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:42.083804 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:42.083743    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8da3844c-03db-451b-87c0-96f1c62167d9-console-oauth-config\") pod \"console-5f6df9f67-6dfhh\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") " pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:42.084068 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:42.084027    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8da3844c-03db-451b-87c0-96f1c62167d9-console-serving-cert\") pod \"console-5f6df9f67-6dfhh\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") " pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:42.087901 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:42.087859    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zp9d\" (UniqueName: \"kubernetes.io/projected/8da3844c-03db-451b-87c0-96f1c62167d9-kube-api-access-8zp9d\") pod \"console-5f6df9f67-6dfhh\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") " pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:42.182857 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:42.182776    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:42.450523 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:42.450418    5070 patch_prober.go:29] interesting pod/image-registry-67ffc88546-z24dr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 07:53:42.450523 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:42.450502    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-67ffc88546-z24dr" podUID="119141fd-3c4f-4b94-9f18-6892986867e9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 07:53:42.472058 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:42.472023    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f6df9f67-6dfhh"]
Nov 03 07:53:42.476706 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:42.476678    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da3844c_03db_451b_87c0_96f1c62167d9.slice/crio-babcb86d729169ca7fc680c1c925439af5d4158fc3e57c26f6b2cc71ee00b2ff WatchSource:0}: Error finding container babcb86d729169ca7fc680c1c925439af5d4158fc3e57c26f6b2cc71ee00b2ff: Status 404 returned error can't find the container with id babcb86d729169ca7fc680c1c925439af5d4158fc3e57c26f6b2cc71ee00b2ff
Nov 03 07:53:43.244848 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:43.244727    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-l99g7" event={"ID":"bcbcf547-bdcb-46f9-bd2c-9164868a80ec","Type":"ContainerStarted","Data":"bb4128aeb6704fff13ebe75c739fbea4105c2124ae3fcad4ce6c90e84e90a44e"}
Nov 03 07:53:43.244848 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:43.244806    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-l99g7" event={"ID":"bcbcf547-bdcb-46f9-bd2c-9164868a80ec","Type":"ContainerStarted","Data":"46b00b60f969c8259c7b8e2b7fff51ce444e5ddc23897d5fbf00d4a018eb6520"}
Nov 03 07:53:43.246844 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:43.246789    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6df9f67-6dfhh" event={"ID":"8da3844c-03db-451b-87c0-96f1c62167d9","Type":"ContainerStarted","Data":"babcb86d729169ca7fc680c1c925439af5d4158fc3e57c26f6b2cc71ee00b2ff"}
Nov 03 07:53:43.271143 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:43.269841    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-l99g7" podStartSLOduration=1.844214641 podStartE2EDuration="3.269830032s" totalImagesPullingTime="1.425615391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:43.26947482 +0000 UTC m=+21.383938753" watchObservedRunningTime="2026-11-03 07:53:43.269830032 +0000 UTC m=+21.384293963"
Nov 03 07:53:43.295789 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:43.294333    5070 patch_prober.go:29] interesting pod/image-registry-67ffc88546-z24dr container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Nov 03 07:53:43.295789 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:43.294386    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-67ffc88546-z24dr" podUID="119141fd-3c4f-4b94-9f18-6892986867e9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 07:53:45.567259 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.567226    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4sbrg"]
Nov 03 07:53:45.588481 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.587643    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.613680 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.613622    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 07:53:45.613861 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.613828    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 07:53:45.614571 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.614526    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8t4tj\""
Nov 03 07:53:45.658628 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.658596    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 07:53:45.714922 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.714890    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60772c10-1acb-489a-89bc-14388a92c001-sys\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.715092 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.714929    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60772c10-1acb-489a-89bc-14388a92c001-node-exporter-tls\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.715092 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.714960    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgqvd\" (UniqueName: \"kubernetes.io/projected/60772c10-1acb-489a-89bc-14388a92c001-kube-api-access-mgqvd\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.715092 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.714990    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/60772c10-1acb-489a-89bc-14388a92c001-node-exporter-accelerators-collector-config\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.715092 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.715018    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60772c10-1acb-489a-89bc-14388a92c001-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.715092 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.715044    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/60772c10-1acb-489a-89bc-14388a92c001-root\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.715092 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.715078    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/60772c10-1acb-489a-89bc-14388a92c001-node-exporter-wtmp\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.715411 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.715117    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/60772c10-1acb-489a-89bc-14388a92c001-node-exporter-textfile\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.715411 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.715179    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60772c10-1acb-489a-89bc-14388a92c001-metrics-client-ca\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.816313 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.816278    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/60772c10-1acb-489a-89bc-14388a92c001-node-exporter-textfile\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.816481 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.816456    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60772c10-1acb-489a-89bc-14388a92c001-metrics-client-ca\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.816572 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.816553    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60772c10-1acb-489a-89bc-14388a92c001-sys\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.816632 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.816588    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60772c10-1acb-489a-89bc-14388a92c001-node-exporter-tls\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.816632 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.816616    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgqvd\" (UniqueName: \"kubernetes.io/projected/60772c10-1acb-489a-89bc-14388a92c001-kube-api-access-mgqvd\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.816735 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.816647    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/60772c10-1acb-489a-89bc-14388a92c001-node-exporter-accelerators-collector-config\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.816735 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.816653    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/60772c10-1acb-489a-89bc-14388a92c001-node-exporter-textfile\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.816735 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.816676    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60772c10-1acb-489a-89bc-14388a92c001-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.816896 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.816740    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/60772c10-1acb-489a-89bc-14388a92c001-root\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.816896 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.816809    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/60772c10-1acb-489a-89bc-14388a92c001-node-exporter-wtmp\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.817040 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.816996    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/60772c10-1acb-489a-89bc-14388a92c001-node-exporter-wtmp\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.817162 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.817064    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60772c10-1acb-489a-89bc-14388a92c001-sys\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.817162 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.817090    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60772c10-1acb-489a-89bc-14388a92c001-metrics-client-ca\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.817503 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.817448    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/60772c10-1acb-489a-89bc-14388a92c001-node-exporter-accelerators-collector-config\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.817503 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.817475    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/60772c10-1acb-489a-89bc-14388a92c001-root\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.819526 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.819500    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60772c10-1acb-489a-89bc-14388a92c001-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.819662 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.819644    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60772c10-1acb-489a-89bc-14388a92c001-node-exporter-tls\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.823088 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.823035    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgqvd\" (UniqueName: \"kubernetes.io/projected/60772c10-1acb-489a-89bc-14388a92c001-kube-api-access-mgqvd\") pod \"node-exporter-4sbrg\" (UID: \"60772c10-1acb-489a-89bc-14388a92c001\") " pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:45.911165 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.911137    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9j2f9"
Nov 03 07:53:45.971753 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:45.971566    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4sbrg"
Nov 03 07:53:46.012085 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:46.012057    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60772c10_1acb_489a_89bc_14388a92c001.slice/crio-2ee190e7219bf5fb47df65d29d4728121c04e5d49dcbc545452ea8f1e0e8465f WatchSource:0}: Error finding container 2ee190e7219bf5fb47df65d29d4728121c04e5d49dcbc545452ea8f1e0e8465f: Status 404 returned error can't find the container with id 2ee190e7219bf5fb47df65d29d4728121c04e5d49dcbc545452ea8f1e0e8465f
Nov 03 07:53:46.262693 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:46.262652    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4sbrg" event={"ID":"60772c10-1acb-489a-89bc-14388a92c001","Type":"ContainerStarted","Data":"2ee190e7219bf5fb47df65d29d4728121c04e5d49dcbc545452ea8f1e0e8465f"}
Nov 03 07:53:46.977439 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:46.977406    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:53:47.270331 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:47.270251    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6df9f67-6dfhh" event={"ID":"8da3844c-03db-451b-87c0-96f1c62167d9","Type":"ContainerStarted","Data":"b82b1f8696ece0e83101481c4adff477ede3670fe8936b0cd1576770fe9352e7"}
Nov 03 07:53:48.276174 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:48.276141    5070 generic.go:356] "Generic (PLEG): container finished" podID="60772c10-1acb-489a-89bc-14388a92c001" containerID="6c6c166d1a2e79df4b70a88913c5f3b88020f714a940c616986ff9e71cc0ff29" exitCode=0
Nov 03 07:53:48.276576 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:48.276227    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4sbrg" event={"ID":"60772c10-1acb-489a-89bc-14388a92c001","Type":"ContainerDied","Data":"6c6c166d1a2e79df4b70a88913c5f3b88020f714a940c616986ff9e71cc0ff29"}
Nov 03 07:53:48.292535 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:48.292494    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5f6df9f67-6dfhh" podStartSLOduration=3.2507920439999998 podStartE2EDuration="7.292481268s" totalImagesPullingTime="4.041689224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:47.287443687 +0000 UTC m=+25.401907619" watchObservedRunningTime="2026-11-03 07:53:48.292481268 +0000 UTC m=+26.406945199"
Nov 03 07:53:49.283668 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:49.283630    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4sbrg" event={"ID":"60772c10-1acb-489a-89bc-14388a92c001","Type":"ContainerStarted","Data":"0400918d37630b3c608b31dea2b472a7580340951da5433a4b597e2ccd3eed43"}
Nov 03 07:53:49.284108 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:49.283691    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4sbrg" event={"ID":"60772c10-1acb-489a-89bc-14388a92c001","Type":"ContainerStarted","Data":"ba76da22fb69c85a9198657928113ba0f84d2f237c883e27e4b4e2d3bcbde202"}
Nov 03 07:53:49.299696 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:49.299647    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4sbrg" podStartSLOduration=2.416433064 podStartE2EDuration="4.299633422s" totalImagesPullingTime="1.788235451s" totalInitContainerRuntime="94.964907ms" isStatefulPod=true podCreationTimestamp="2026-11-03 07:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:49.299390247 +0000 UTC m=+27.413854179" watchObservedRunningTime="2026-11-03 07:53:49.299633422 +0000 UTC m=+27.414097355"
Nov 03 07:53:50.859679 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:50.859648    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-c6bms"
Nov 03 07:53:52.058825 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:53:52.058784    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 07:53:52.183547 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:52.183517    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:52.183707 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:52.183572    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:52.190182 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:52.190154    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:52.310303 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:52.310053    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:53:52.450884 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:52.450844    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:53:53.076103 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.076070    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c9975d7d-ctzrz"]
Nov 03 07:53:53.109209 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.109175    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9975d7d-ctzrz"]
Nov 03 07:53:53.109382 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.109313    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.113815 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.113772    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 07:53:53.294678 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.294643    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-tmp\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.294880 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.294748    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-trusted-ca-bundle\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.294880 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.294811    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-console-config\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.294880 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.294830    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-console-oauth-config\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.294880 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.294844    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz7m6\" (UniqueName: \"kubernetes.io/projected/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-kube-api-access-xz7m6\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.294880 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.294870    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-console-serving-cert\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.295216 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.295055    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-service-ca\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.295216 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.295122    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-oauth-serving-cert\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.396725 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.396690    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-trusted-ca-bundle\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.396925 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.396747    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-console-config\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.396925 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.396804    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-console-oauth-config\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.397024 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.396918    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz7m6\" (UniqueName: \"kubernetes.io/projected/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-kube-api-access-xz7m6\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.397024 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.396954    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-console-serving-cert\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.397024 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.396990    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-service-ca\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.397177 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.397024    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-oauth-serving-cert\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.397177 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.397053    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-tmp\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.397550 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.397524    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-console-config\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.397744 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.397720    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-tmp\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.398217 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.398199    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-service-ca\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.402729 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.398392    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-trusted-ca-bundle\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.402729 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.398886    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-oauth-serving-cert\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.402729 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.400652    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-console-oauth-config\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.404735 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.404712    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz7m6\" (UniqueName: \"kubernetes.io/projected/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-kube-api-access-xz7m6\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.405390 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.405371    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-console-serving-cert\") pod \"console-7c9975d7d-ctzrz\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") " pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.444853 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.444828    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:53:53.755610 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:53.755579    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c9975d7d-ctzrz"]
Nov 03 07:53:53.758165 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:53.758137    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fbc88d3_6002_4ab2_beed_7cd2c55ec205.slice/crio-1d55a490f7f70b19a68939d4f52aa0011f2341635661e64bad87ec18a39a1328 WatchSource:0}: Error finding container 1d55a490f7f70b19a68939d4f52aa0011f2341635661e64bad87ec18a39a1328: Status 404 returned error can't find the container with id 1d55a490f7f70b19a68939d4f52aa0011f2341635661e64bad87ec18a39a1328
Nov 03 07:53:54.316241 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:54.316205    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9975d7d-ctzrz" event={"ID":"5fbc88d3-6002-4ab2-beed-7cd2c55ec205","Type":"ContainerStarted","Data":"a6dde7b7ea09fc3c178b0359c4c41d4fdf14e092dff23be643416d6b98892501"}
Nov 03 07:53:54.316688 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:54.316256    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9975d7d-ctzrz" event={"ID":"5fbc88d3-6002-4ab2-beed-7cd2c55ec205","Type":"ContainerStarted","Data":"1d55a490f7f70b19a68939d4f52aa0011f2341635661e64bad87ec18a39a1328"}
Nov 03 07:53:54.333023 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:54.332955    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7c9975d7d-ctzrz" podStartSLOduration=1.332940099 podStartE2EDuration="1.332940099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:54.33266138 +0000 UTC m=+32.447125311" watchObservedRunningTime="2026-11-03 07:53:54.332940099 +0000 UTC m=+32.447404033"
Nov 03 07:53:55.483920 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.483879    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c9975d7d-ctzrz"]
Nov 03 07:53:55.507497 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.507466    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f8b567755-rww99"]
Nov 03 07:53:55.538988 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.538962    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f8b567755-rww99"]
Nov 03 07:53:55.539151 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.539083    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.721314 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.721278    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b395996-50fb-416f-a744-fffab9e943f9-tmp\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.721468 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.721331    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-console-config\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.721468 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.721426    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-oauth-serving-cert\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.721555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.721484    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b395996-50fb-416f-a744-fffab9e943f9-console-serving-cert\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.721555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.721533    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b395996-50fb-416f-a744-fffab9e943f9-console-oauth-config\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.721637 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.721565    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlx9t\" (UniqueName: \"kubernetes.io/projected/3b395996-50fb-416f-a744-fffab9e943f9-kube-api-access-qlx9t\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.721694 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.721673    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-service-ca\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.721728 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.721711    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-trusted-ca-bundle\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.822824 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.822752    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b395996-50fb-416f-a744-fffab9e943f9-console-serving-cert\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.822824 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.822791    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b395996-50fb-416f-a744-fffab9e943f9-console-oauth-config\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.822824 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.822807    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlx9t\" (UniqueName: \"kubernetes.io/projected/3b395996-50fb-416f-a744-fffab9e943f9-kube-api-access-qlx9t\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.823089 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.822851    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-service-ca\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.823089 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.822884    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-trusted-ca-bundle\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.823089 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.822934    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b395996-50fb-416f-a744-fffab9e943f9-tmp\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.823089 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.823067    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-console-config\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.823292 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.823156    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-oauth-serving-cert\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.823371 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.823343    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b395996-50fb-416f-a744-fffab9e943f9-tmp\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.823726 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.823698    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-service-ca\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.823887 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.823820    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-console-config\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.824056 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.824032    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-trusted-ca-bundle\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.824128 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.824032    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-oauth-serving-cert\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.825289 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.825267    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b395996-50fb-416f-a744-fffab9e943f9-console-serving-cert\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.825289 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.825283    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b395996-50fb-416f-a744-fffab9e943f9-console-oauth-config\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.829274 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.829253    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlx9t\" (UniqueName: \"kubernetes.io/projected/3b395996-50fb-416f-a744-fffab9e943f9-kube-api-access-qlx9t\") pod \"console-6f8b567755-rww99\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") " pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:55.872723 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:55.872695    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:53:56.161650 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:56.161618    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f8b567755-rww99"]
Nov 03 07:53:56.164462 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:53:56.164434    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b395996_50fb_416f_a744_fffab9e943f9.slice/crio-6bdba2c53ad4835b4959da433e38bbde62e378039a9be1506d17e7f08605bfeb WatchSource:0}: Error finding container 6bdba2c53ad4835b4959da433e38bbde62e378039a9be1506d17e7f08605bfeb: Status 404 returned error can't find the container with id 6bdba2c53ad4835b4959da433e38bbde62e378039a9be1506d17e7f08605bfeb
Nov 03 07:53:56.331519 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:56.331485    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f8b567755-rww99" event={"ID":"3b395996-50fb-416f-a744-fffab9e943f9","Type":"ContainerStarted","Data":"89a74c07fc8ff4cf1d364480e57d9a1c1613f6d15cd42c9ecd4116c1d102f618"}
Nov 03 07:53:56.331708 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:56.331542    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f8b567755-rww99" event={"ID":"3b395996-50fb-416f-a744-fffab9e943f9","Type":"ContainerStarted","Data":"6bdba2c53ad4835b4959da433e38bbde62e378039a9be1506d17e7f08605bfeb"}
Nov 03 07:53:56.333126 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:56.333096    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-qmpgn" event={"ID":"5400462b-2298-41e2-b44b-caa2f3a1e496","Type":"ContainerStarted","Data":"e0c628bb5dfd40ee49b59fb8e3238f3386d13f119103d481b16f2e7ed001aed8"}
Nov 03 07:53:56.333219 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:56.333153    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-qmpgn"
Nov 03 07:53:56.334482 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:56.334458    5070 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-qmpgn container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/\": dial tcp 10.132.0.29:8080: connect: connection refused" start-of-body=
Nov 03 07:53:56.334590 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:56.334504    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-qmpgn" podUID="5400462b-2298-41e2-b44b-caa2f3a1e496" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": dial tcp 10.132.0.29:8080: connect: connection refused"
Nov 03 07:53:56.349024 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:56.348954    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6f8b567755-rww99" podStartSLOduration=1.348936087 podStartE2EDuration="1.348936087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:56.347911247 +0000 UTC m=+34.462375179" watchObservedRunningTime="2026-11-03 07:53:56.348936087 +0000 UTC m=+34.463400017"
Nov 03 07:53:56.367432 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:56.367397    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-qmpgn" podStartSLOduration=1.769056833 podStartE2EDuration="20.367385043s" totalImagesPullingTime="18.59832821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:53:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:53:56.36705185 +0000 UTC m=+34.481515782" watchObservedRunningTime="2026-11-03 07:53:56.367385043 +0000 UTC m=+34.481848968"
Nov 03 07:53:57.349631 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:57.349599    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-qmpgn"
Nov 03 07:53:59.225194 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:59.225150    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 07:53:59.278548 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:53:59.270817    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67ffc88546-z24dr"]
Nov 03 07:54:01.996837 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:01.996794    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-74774885c5-97jxz" podUID="df604a27-2a06-42a0-bcbd-f7a10c1420c1" containerName="registry" containerID="cri-o://a3024eb7124778687f5cfaace0f7b5eebe83f31768c33bc3b423769497a90de9" gracePeriod=30
Nov 03 07:54:02.315957 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.315932    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:54:02.382909 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.382873    5070 generic.go:356] "Generic (PLEG): container finished" podID="df604a27-2a06-42a0-bcbd-f7a10c1420c1" containerID="a3024eb7124778687f5cfaace0f7b5eebe83f31768c33bc3b423769497a90de9" exitCode=0
Nov 03 07:54:02.383086 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.382928    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74774885c5-97jxz"
Nov 03 07:54:02.383086 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.382971    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74774885c5-97jxz" event={"ID":"df604a27-2a06-42a0-bcbd-f7a10c1420c1","Type":"ContainerDied","Data":"a3024eb7124778687f5cfaace0f7b5eebe83f31768c33bc3b423769497a90de9"}
Nov 03 07:54:02.383086 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.383032    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74774885c5-97jxz" event={"ID":"df604a27-2a06-42a0-bcbd-f7a10c1420c1","Type":"ContainerDied","Data":"9f593dedc09571c3546d8760b1f3bddea3be656bde40c0b09e367494b2c33392"}
Nov 03 07:54:02.383086 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.383054    5070 scope.go:122] "RemoveContainer" containerID="a3024eb7124778687f5cfaace0f7b5eebe83f31768c33bc3b423769497a90de9"
Nov 03 07:54:02.401009 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.400986    5070 scope.go:122] "RemoveContainer" containerID="a3024eb7124778687f5cfaace0f7b5eebe83f31768c33bc3b423769497a90de9"
Nov 03 07:54:02.401321 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:54:02.401302    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3024eb7124778687f5cfaace0f7b5eebe83f31768c33bc3b423769497a90de9\": container with ID starting with a3024eb7124778687f5cfaace0f7b5eebe83f31768c33bc3b423769497a90de9 not found: ID does not exist" containerID="a3024eb7124778687f5cfaace0f7b5eebe83f31768c33bc3b423769497a90de9"
Nov 03 07:54:02.401404 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.401333    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3024eb7124778687f5cfaace0f7b5eebe83f31768c33bc3b423769497a90de9"} err="failed to get container status \"a3024eb7124778687f5cfaace0f7b5eebe83f31768c33bc3b423769497a90de9\": rpc error: code = NotFound desc = could not find container \"a3024eb7124778687f5cfaace0f7b5eebe83f31768c33bc3b423769497a90de9\": container with ID starting with a3024eb7124778687f5cfaace0f7b5eebe83f31768c33bc3b423769497a90de9 not found: ID does not exist"
Nov 03 07:54:02.408465 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.408439    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls\") pod \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") "
Nov 03 07:54:02.408574 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.408500    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df604a27-2a06-42a0-bcbd-f7a10c1420c1-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df604a27-2a06-42a0-bcbd-f7a10c1420c1-ca-trust-extracted\") pod \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") "
Nov 03 07:54:02.408574 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.408550    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df604a27-2a06-42a0-bcbd-f7a10c1420c1-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df604a27-2a06-42a0-bcbd-f7a10c1420c1-installation-pull-secrets\") pod \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") "
Nov 03 07:54:02.408679 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.408590    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-certificates\") pod \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") "
Nov 03 07:54:02.408679 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.408632    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-kube-api-access-4wwkr\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-kube-api-access-4wwkr\") pod \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") "
Nov 03 07:54:02.408679 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.408667    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-bound-sa-token\") pod \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") "
Nov 03 07:54:02.408859 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.408708    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df604a27-2a06-42a0-bcbd-f7a10c1420c1-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df604a27-2a06-42a0-bcbd-f7a10c1420c1-image-registry-private-configuration\") pod \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") "
Nov 03 07:54:02.408859 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.408776    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df604a27-2a06-42a0-bcbd-f7a10c1420c1-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df604a27-2a06-42a0-bcbd-f7a10c1420c1-trusted-ca\") pod \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\" (UID: \"df604a27-2a06-42a0-bcbd-f7a10c1420c1\") "
Nov 03 07:54:02.409598 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.409545    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-certificates" pod "df604a27-2a06-42a0-bcbd-f7a10c1420c1" (UID: "df604a27-2a06-42a0-bcbd-f7a10c1420c1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:54:02.411453 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.411428    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df604a27-2a06-42a0-bcbd-f7a10c1420c1-trusted-ca" pod "df604a27-2a06-42a0-bcbd-f7a10c1420c1" (UID: "df604a27-2a06-42a0-bcbd-f7a10c1420c1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:54:02.411604 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.411493    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-bound-sa-token" pod "df604a27-2a06-42a0-bcbd-f7a10c1420c1" (UID: "df604a27-2a06-42a0-bcbd-f7a10c1420c1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:54:02.412160 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.412135    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls" pod "df604a27-2a06-42a0-bcbd-f7a10c1420c1" (UID: "df604a27-2a06-42a0-bcbd-f7a10c1420c1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:54:02.413442 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.413390    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df604a27-2a06-42a0-bcbd-f7a10c1420c1-image-registry-private-configuration" pod "df604a27-2a06-42a0-bcbd-f7a10c1420c1" (UID: "df604a27-2a06-42a0-bcbd-f7a10c1420c1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:54:02.413634 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.413614    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df604a27-2a06-42a0-bcbd-f7a10c1420c1-installation-pull-secrets" pod "df604a27-2a06-42a0-bcbd-f7a10c1420c1" (UID: "df604a27-2a06-42a0-bcbd-f7a10c1420c1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:54:02.413991 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.413960    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-kube-api-access-4wwkr" pod "df604a27-2a06-42a0-bcbd-f7a10c1420c1" (UID: "df604a27-2a06-42a0-bcbd-f7a10c1420c1"). InnerVolumeSpecName "kube-api-access-4wwkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:54:02.417555 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.417532    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df604a27-2a06-42a0-bcbd-f7a10c1420c1-ca-trust-extracted" pod "df604a27-2a06-42a0-bcbd-f7a10c1420c1" (UID: "df604a27-2a06-42a0-bcbd-f7a10c1420c1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:54:02.509986 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.509957    5070 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-tls\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:02.510117 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.509991    5070 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df604a27-2a06-42a0-bcbd-f7a10c1420c1-ca-trust-extracted\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:02.510117 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.510005    5070 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df604a27-2a06-42a0-bcbd-f7a10c1420c1-installation-pull-secrets\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:02.510117 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.510016    5070 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df604a27-2a06-42a0-bcbd-f7a10c1420c1-registry-certificates\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:02.510117 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.510026    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wwkr\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-kube-api-access-4wwkr\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:02.510117 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.510034    5070 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df604a27-2a06-42a0-bcbd-f7a10c1420c1-bound-sa-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:02.510117 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.510048    5070 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df604a27-2a06-42a0-bcbd-f7a10c1420c1-image-registry-private-configuration\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:02.510117 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.510062    5070 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df604a27-2a06-42a0-bcbd-f7a10c1420c1-trusted-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:02.705386 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.705352    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74774885c5-97jxz"]
Nov 03 07:54:02.710629 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:02.710602    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-74774885c5-97jxz"]
Nov 03 07:54:03.445694 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:03.445657    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:54:04.024895 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:04.024854    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df604a27-2a06-42a0-bcbd-f7a10c1420c1" path="/var/lib/kubelet/pods/df604a27-2a06-42a0-bcbd-f7a10c1420c1/volumes"
Nov 03 07:54:05.873499 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:05.873465    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:54:05.873958 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:05.873540    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:54:05.878083 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:05.878061    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:54:06.406981 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:06.406953    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:54:06.448266 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:06.448232    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f6df9f67-6dfhh"]
Nov 03 07:54:09.320824 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:09.320784    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-l5f6q-fpwdg"]
Nov 03 07:54:09.321390 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:09.320983    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-l5f6q-fpwdg" podUID="66e53c53-b31d-43f7-b7b2-2c90a07dcf09" containerName="insights-gathering" containerID="cri-o://de2e064a851278745f4321cebf8e8272e783ea5b321a5e4ff3ccc6634c7f30e3" gracePeriod=30
Nov 03 07:54:09.657311 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:09.657287    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:54:09.769877 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:09.769836    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-service-ca-bundle\") pod \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") "
Nov 03 07:54:09.770045 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:09.769894    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-kube-api-access-klntp\" (UniqueName: \"kubernetes.io/projected/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-kube-api-access-klntp\") pod \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") "
Nov 03 07:54:09.770045 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:09.770020    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-trusted-ca-bundle\") pod \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") "
Nov 03 07:54:09.770166 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:09.770061    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-archives-path\") pod \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\" (UID: \"66e53c53-b31d-43f7-b7b2-2c90a07dcf09\") "
Nov 03 07:54:09.770286 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:09.770256    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-service-ca-bundle" pod "66e53c53-b31d-43f7-b7b2-2c90a07dcf09" (UID: "66e53c53-b31d-43f7-b7b2-2c90a07dcf09"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:54:09.770372 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:09.770350    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-trusted-ca-bundle" pod "66e53c53-b31d-43f7-b7b2-2c90a07dcf09" (UID: "66e53c53-b31d-43f7-b7b2-2c90a07dcf09"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:54:09.770410 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:09.770395    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-archives-path" pod "66e53c53-b31d-43f7-b7b2-2c90a07dcf09" (UID: "66e53c53-b31d-43f7-b7b2-2c90a07dcf09"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:54:09.772349 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:09.772319    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-kube-api-access-klntp" pod "66e53c53-b31d-43f7-b7b2-2c90a07dcf09" (UID: "66e53c53-b31d-43f7-b7b2-2c90a07dcf09"). InnerVolumeSpecName "kube-api-access-klntp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:54:09.871080 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:09.871052    5070 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-trusted-ca-bundle\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:09.871080 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:09.871077    5070 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-archives-path\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:09.871233 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:09.871086    5070 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-service-ca-bundle\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:09.871233 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:09.871096    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klntp\" (UniqueName: \"kubernetes.io/projected/66e53c53-b31d-43f7-b7b2-2c90a07dcf09-kube-api-access-klntp\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:10.430638 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:10.430604    5070 generic.go:356] "Generic (PLEG): container finished" podID="66e53c53-b31d-43f7-b7b2-2c90a07dcf09" containerID="de2e064a851278745f4321cebf8e8272e783ea5b321a5e4ff3ccc6634c7f30e3" exitCode=2
Nov 03 07:54:10.431026 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:10.430646    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-l5f6q-fpwdg"
Nov 03 07:54:10.431026 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:10.430655    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-l5f6q-fpwdg" event={"ID":"66e53c53-b31d-43f7-b7b2-2c90a07dcf09","Type":"ContainerDied","Data":"de2e064a851278745f4321cebf8e8272e783ea5b321a5e4ff3ccc6634c7f30e3"}
Nov 03 07:54:10.431026 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:10.430702    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-l5f6q-fpwdg" event={"ID":"66e53c53-b31d-43f7-b7b2-2c90a07dcf09","Type":"ContainerDied","Data":"9c0e481f1c0afebf976b20aa412a47bc61e96b6a980d3ca65fdde537b1ff35ce"}
Nov 03 07:54:10.431026 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:10.430724    5070 scope.go:122] "RemoveContainer" containerID="de2e064a851278745f4321cebf8e8272e783ea5b321a5e4ff3ccc6634c7f30e3"
Nov 03 07:54:10.439218 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:10.439199    5070 scope.go:122] "RemoveContainer" containerID="de2e064a851278745f4321cebf8e8272e783ea5b321a5e4ff3ccc6634c7f30e3"
Nov 03 07:54:10.439490 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:54:10.439474    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de2e064a851278745f4321cebf8e8272e783ea5b321a5e4ff3ccc6634c7f30e3\": container with ID starting with de2e064a851278745f4321cebf8e8272e783ea5b321a5e4ff3ccc6634c7f30e3 not found: ID does not exist" containerID="de2e064a851278745f4321cebf8e8272e783ea5b321a5e4ff3ccc6634c7f30e3"
Nov 03 07:54:10.439570 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:10.439505    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de2e064a851278745f4321cebf8e8272e783ea5b321a5e4ff3ccc6634c7f30e3"} err="failed to get container status \"de2e064a851278745f4321cebf8e8272e783ea5b321a5e4ff3ccc6634c7f30e3\": rpc error: code = NotFound desc = could not find container \"de2e064a851278745f4321cebf8e8272e783ea5b321a5e4ff3ccc6634c7f30e3\": container with ID starting with de2e064a851278745f4321cebf8e8272e783ea5b321a5e4ff3ccc6634c7f30e3 not found: ID does not exist"
Nov 03 07:54:10.443078 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:10.443055    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-l5f6q-fpwdg"]
Nov 03 07:54:10.449034 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:10.449016    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-l5f6q-fpwdg"]
Nov 03 07:54:10.942408 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:10.942372    5070 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9j2f9_676416b5-105e-4650-baed-92baf169a9a6/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:11.079294 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:11.079255    5070 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9j2f9_676416b5-105e-4650-baed-92baf169a9a6/konnectivity-agent/0.log" limit=1024
Nov 03 07:54:12.023246 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:12.023218    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66e53c53-b31d-43f7-b7b2-2c90a07dcf09" path="/var/lib/kubelet/pods/66e53c53-b31d-43f7-b7b2-2c90a07dcf09/volumes"
Nov 03 07:54:21.353866 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.353799    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7c9975d7d-ctzrz" podUID="5fbc88d3-6002-4ab2-beed-7cd2c55ec205" containerName="console" containerID="cri-o://a6dde7b7ea09fc3c178b0359c4c41d4fdf14e092dff23be643416d6b98892501" gracePeriod=15
Nov 03 07:54:21.486359 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.486318    5070 generic.go:356] "Generic (PLEG): container finished" podID="5fbc88d3-6002-4ab2-beed-7cd2c55ec205" containerID="a6dde7b7ea09fc3c178b0359c4c41d4fdf14e092dff23be643416d6b98892501" exitCode=0
Nov 03 07:54:21.486520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.486388    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9975d7d-ctzrz" event={"ID":"5fbc88d3-6002-4ab2-beed-7cd2c55ec205","Type":"ContainerDied","Data":"a6dde7b7ea09fc3c178b0359c4c41d4fdf14e092dff23be643416d6b98892501"}
Nov 03 07:54:21.609552 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.609496    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:54:21.663901 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.663877    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-trusted-ca-bundle\") pod \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") "
Nov 03 07:54:21.664029 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.663914    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-oauth-serving-cert\") pod \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") "
Nov 03 07:54:21.664029 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.663947    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-kube-api-access-xz7m6\" (UniqueName: \"kubernetes.io/projected/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-kube-api-access-xz7m6\") pod \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") "
Nov 03 07:54:21.664029 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.664009    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-console-serving-cert\") pod \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") "
Nov 03 07:54:21.664164 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.664062    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-tmp\") pod \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") "
Nov 03 07:54:21.664164 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.664110    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-service-ca\") pod \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") "
Nov 03 07:54:21.664164 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.664155    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-console-config\" (UniqueName: \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-console-config\") pod \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") "
Nov 03 07:54:21.664456 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.664182    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-console-oauth-config\") pod \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\" (UID: \"5fbc88d3-6002-4ab2-beed-7cd2c55ec205\") "
Nov 03 07:54:21.664456 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.664309    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-oauth-serving-cert" pod "5fbc88d3-6002-4ab2-beed-7cd2c55ec205" (UID: "5fbc88d3-6002-4ab2-beed-7cd2c55ec205"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:54:21.664456 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.664346    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-tmp" pod "5fbc88d3-6002-4ab2-beed-7cd2c55ec205" (UID: "5fbc88d3-6002-4ab2-beed-7cd2c55ec205"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:54:21.664456 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.664359    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-trusted-ca-bundle" pod "5fbc88d3-6002-4ab2-beed-7cd2c55ec205" (UID: "5fbc88d3-6002-4ab2-beed-7cd2c55ec205"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:54:21.664667 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.664491    5070 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-tmp\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:21.664667 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.664507    5070 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-trusted-ca-bundle\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:21.664667 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.664517    5070 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-oauth-serving-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:21.664847 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.664723    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-service-ca" pod "5fbc88d3-6002-4ab2-beed-7cd2c55ec205" (UID: "5fbc88d3-6002-4ab2-beed-7cd2c55ec205"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:54:21.664924 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.664900    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-console-config" pod "5fbc88d3-6002-4ab2-beed-7cd2c55ec205" (UID: "5fbc88d3-6002-4ab2-beed-7cd2c55ec205"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:54:21.666466 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.666444    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-kube-api-access-xz7m6" pod "5fbc88d3-6002-4ab2-beed-7cd2c55ec205" (UID: "5fbc88d3-6002-4ab2-beed-7cd2c55ec205"). InnerVolumeSpecName "kube-api-access-xz7m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:54:21.666549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.666527    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-console-serving-cert" pod "5fbc88d3-6002-4ab2-beed-7cd2c55ec205" (UID: "5fbc88d3-6002-4ab2-beed-7cd2c55ec205"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:54:21.666549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.666541    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-console-oauth-config" pod "5fbc88d3-6002-4ab2-beed-7cd2c55ec205" (UID: "5fbc88d3-6002-4ab2-beed-7cd2c55ec205"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:54:21.765087 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.765054    5070 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-console-serving-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:21.765087 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.765087    5070 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-service-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:21.765087 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.765097    5070 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-console-config\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:21.765258 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.765107    5070 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-console-oauth-config\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:21.765258 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:21.765117    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz7m6\" (UniqueName: \"kubernetes.io/projected/5fbc88d3-6002-4ab2-beed-7cd2c55ec205-kube-api-access-xz7m6\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:22.049793 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:54:22.049750    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 07:54:22.491975 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:22.491946    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c9975d7d-ctzrz"
Nov 03 07:54:22.492345 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:22.491945    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c9975d7d-ctzrz" event={"ID":"5fbc88d3-6002-4ab2-beed-7cd2c55ec205","Type":"ContainerDied","Data":"1d55a490f7f70b19a68939d4f52aa0011f2341635661e64bad87ec18a39a1328"}
Nov 03 07:54:22.492345 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:22.492083    5070 scope.go:122] "RemoveContainer" containerID="a6dde7b7ea09fc3c178b0359c4c41d4fdf14e092dff23be643416d6b98892501"
Nov 03 07:54:22.505166 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:22.505142    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c9975d7d-ctzrz"]
Nov 03 07:54:22.511228 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:22.511206    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c9975d7d-ctzrz"]
Nov 03 07:54:24.025090 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.025044    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fbc88d3-6002-4ab2-beed-7cd2c55ec205" path="/var/lib/kubelet/pods/5fbc88d3-6002-4ab2-beed-7cd2c55ec205/volumes"
Nov 03 07:54:24.310876 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.310803    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-67ffc88546-z24dr" podUID="119141fd-3c4f-4b94-9f18-6892986867e9" containerName="registry" containerID="cri-o://aaf08aaf001f1fdc49cba51897c692024c67ea83d9a269af54616433a10c309b" gracePeriod=30
Nov 03 07:54:24.504165 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.504133    5070 generic.go:356] "Generic (PLEG): container finished" podID="119141fd-3c4f-4b94-9f18-6892986867e9" containerID="aaf08aaf001f1fdc49cba51897c692024c67ea83d9a269af54616433a10c309b" exitCode=0
Nov 03 07:54:24.504322 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.504206    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67ffc88546-z24dr" event={"ID":"119141fd-3c4f-4b94-9f18-6892986867e9","Type":"ContainerDied","Data":"aaf08aaf001f1fdc49cba51897c692024c67ea83d9a269af54616433a10c309b"}
Nov 03 07:54:24.592852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.592797    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:54:24.689259 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.689232    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-bound-sa-token\") pod \"119141fd-3c4f-4b94-9f18-6892986867e9\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") "
Nov 03 07:54:24.689370 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.689275    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-kube-api-access-vkn8t\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-kube-api-access-vkn8t\") pod \"119141fd-3c4f-4b94-9f18-6892986867e9\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") "
Nov 03 07:54:24.689370 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.689305    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls\") pod \"119141fd-3c4f-4b94-9f18-6892986867e9\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") "
Nov 03 07:54:24.689370 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.689329    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/119141fd-3c4f-4b94-9f18-6892986867e9-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/119141fd-3c4f-4b94-9f18-6892986867e9-registry-certificates\") pod \"119141fd-3c4f-4b94-9f18-6892986867e9\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") "
Nov 03 07:54:24.689370 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.689352    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/119141fd-3c4f-4b94-9f18-6892986867e9-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/119141fd-3c4f-4b94-9f18-6892986867e9-image-registry-private-configuration\") pod \"119141fd-3c4f-4b94-9f18-6892986867e9\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") "
Nov 03 07:54:24.689539 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.689443    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/119141fd-3c4f-4b94-9f18-6892986867e9-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/119141fd-3c4f-4b94-9f18-6892986867e9-trusted-ca\") pod \"119141fd-3c4f-4b94-9f18-6892986867e9\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") "
Nov 03 07:54:24.689539 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.689469    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/119141fd-3c4f-4b94-9f18-6892986867e9-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/119141fd-3c4f-4b94-9f18-6892986867e9-installation-pull-secrets\") pod \"119141fd-3c4f-4b94-9f18-6892986867e9\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") "
Nov 03 07:54:24.689539 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.689503    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119141fd-3c4f-4b94-9f18-6892986867e9-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/119141fd-3c4f-4b94-9f18-6892986867e9-ca-trust-extracted\") pod \"119141fd-3c4f-4b94-9f18-6892986867e9\" (UID: \"119141fd-3c4f-4b94-9f18-6892986867e9\") "
Nov 03 07:54:24.689868 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.689835    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/119141fd-3c4f-4b94-9f18-6892986867e9-registry-certificates" pod "119141fd-3c4f-4b94-9f18-6892986867e9" (UID: "119141fd-3c4f-4b94-9f18-6892986867e9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:54:24.690201 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.690178    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/119141fd-3c4f-4b94-9f18-6892986867e9-trusted-ca" pod "119141fd-3c4f-4b94-9f18-6892986867e9" (UID: "119141fd-3c4f-4b94-9f18-6892986867e9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:54:24.692124 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.692096    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-kube-api-access-vkn8t" pod "119141fd-3c4f-4b94-9f18-6892986867e9" (UID: "119141fd-3c4f-4b94-9f18-6892986867e9"). InnerVolumeSpecName "kube-api-access-vkn8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:54:24.692244 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.692132    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/119141fd-3c4f-4b94-9f18-6892986867e9-image-registry-private-configuration" pod "119141fd-3c4f-4b94-9f18-6892986867e9" (UID: "119141fd-3c4f-4b94-9f18-6892986867e9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:54:24.692356 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.692336    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/119141fd-3c4f-4b94-9f18-6892986867e9-installation-pull-secrets" pod "119141fd-3c4f-4b94-9f18-6892986867e9" (UID: "119141fd-3c4f-4b94-9f18-6892986867e9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:54:24.692411 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.692380    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-bound-sa-token" pod "119141fd-3c4f-4b94-9f18-6892986867e9" (UID: "119141fd-3c4f-4b94-9f18-6892986867e9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:54:24.692411 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.692404    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls" pod "119141fd-3c4f-4b94-9f18-6892986867e9" (UID: "119141fd-3c4f-4b94-9f18-6892986867e9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:54:24.699592 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.699566    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119141fd-3c4f-4b94-9f18-6892986867e9-ca-trust-extracted" pod "119141fd-3c4f-4b94-9f18-6892986867e9" (UID: "119141fd-3c4f-4b94-9f18-6892986867e9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:54:24.790972 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.790953    5070 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/119141fd-3c4f-4b94-9f18-6892986867e9-trusted-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:24.791060 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.790972    5070 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/119141fd-3c4f-4b94-9f18-6892986867e9-installation-pull-secrets\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:24.791060 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.790984    5070 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/119141fd-3c4f-4b94-9f18-6892986867e9-ca-trust-extracted\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:24.791060 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.790993    5070 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-bound-sa-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:24.791060 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.791003    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkn8t\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-kube-api-access-vkn8t\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:24.791060 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.791012    5070 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/119141fd-3c4f-4b94-9f18-6892986867e9-registry-tls\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:24.791060 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.791021    5070 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/119141fd-3c4f-4b94-9f18-6892986867e9-registry-certificates\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:24.791060 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:24.791030    5070 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/119141fd-3c4f-4b94-9f18-6892986867e9-image-registry-private-configuration\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:25.510369 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:25.510339    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67ffc88546-z24dr" event={"ID":"119141fd-3c4f-4b94-9f18-6892986867e9","Type":"ContainerDied","Data":"4c629640eedaa3dd010c0fa7bc1ea73eb5773845c207a0f246ad5b208ffcd924"}
Nov 03 07:54:25.510369 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:25.510360    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67ffc88546-z24dr"
Nov 03 07:54:25.510831 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:25.510394    5070 scope.go:122] "RemoveContainer" containerID="aaf08aaf001f1fdc49cba51897c692024c67ea83d9a269af54616433a10c309b"
Nov 03 07:54:25.528961 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:25.528908    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-67ffc88546-z24dr"]
Nov 03 07:54:25.532539 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:25.532516    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-67ffc88546-z24dr"]
Nov 03 07:54:26.024001 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:26.023971    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="119141fd-3c4f-4b94-9f18-6892986867e9" path="/var/lib/kubelet/pods/119141fd-3c4f-4b94-9f18-6892986867e9/volumes"
Nov 03 07:54:31.470129 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.470073    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5f6df9f67-6dfhh" podUID="8da3844c-03db-451b-87c0-96f1c62167d9" containerName="console" containerID="cri-o://b82b1f8696ece0e83101481c4adff477ede3670fe8936b0cd1576770fe9352e7" gracePeriod=15
Nov 03 07:54:31.760206 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.760184    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:54:31.844837 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.844808    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8da3844c-03db-451b-87c0-96f1c62167d9-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8da3844c-03db-451b-87c0-96f1c62167d9-console-oauth-config\") pod \"8da3844c-03db-451b-87c0-96f1c62167d9\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") "
Nov 03 07:54:31.844962 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.844868    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8da3844c-03db-451b-87c0-96f1c62167d9-kube-api-access-8zp9d\" (UniqueName: \"kubernetes.io/projected/8da3844c-03db-451b-87c0-96f1c62167d9-kube-api-access-8zp9d\") pod \"8da3844c-03db-451b-87c0-96f1c62167d9\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") "
Nov 03 07:54:31.844962 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.844886    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8da3844c-03db-451b-87c0-96f1c62167d9-console-config\" (UniqueName: \"kubernetes.io/configmap/8da3844c-03db-451b-87c0-96f1c62167d9-console-config\") pod \"8da3844c-03db-451b-87c0-96f1c62167d9\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") "
Nov 03 07:54:31.845048 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.844995    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8da3844c-03db-451b-87c0-96f1c62167d9-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8da3844c-03db-451b-87c0-96f1c62167d9-console-serving-cert\") pod \"8da3844c-03db-451b-87c0-96f1c62167d9\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") "
Nov 03 07:54:31.845107 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.845042    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8da3844c-03db-451b-87c0-96f1c62167d9-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8da3844c-03db-451b-87c0-96f1c62167d9-oauth-serving-cert\") pod \"8da3844c-03db-451b-87c0-96f1c62167d9\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") "
Nov 03 07:54:31.845107 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.845085    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8da3844c-03db-451b-87c0-96f1c62167d9-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da3844c-03db-451b-87c0-96f1c62167d9-service-ca\") pod \"8da3844c-03db-451b-87c0-96f1c62167d9\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") "
Nov 03 07:54:31.845215 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.845146    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da3844c-03db-451b-87c0-96f1c62167d9-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8da3844c-03db-451b-87c0-96f1c62167d9-tmp\") pod \"8da3844c-03db-451b-87c0-96f1c62167d9\" (UID: \"8da3844c-03db-451b-87c0-96f1c62167d9\") "
Nov 03 07:54:31.845273 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.845221    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da3844c-03db-451b-87c0-96f1c62167d9-console-config" pod "8da3844c-03db-451b-87c0-96f1c62167d9" (UID: "8da3844c-03db-451b-87c0-96f1c62167d9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:54:31.845429 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.845407    5070 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8da3844c-03db-451b-87c0-96f1c62167d9-console-config\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:31.845517 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.845442    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da3844c-03db-451b-87c0-96f1c62167d9-oauth-serving-cert" pod "8da3844c-03db-451b-87c0-96f1c62167d9" (UID: "8da3844c-03db-451b-87c0-96f1c62167d9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:54:31.845577 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.845530    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da3844c-03db-451b-87c0-96f1c62167d9-tmp" pod "8da3844c-03db-451b-87c0-96f1c62167d9" (UID: "8da3844c-03db-451b-87c0-96f1c62167d9"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:54:31.845577 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.845544    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da3844c-03db-451b-87c0-96f1c62167d9-service-ca" pod "8da3844c-03db-451b-87c0-96f1c62167d9" (UID: "8da3844c-03db-451b-87c0-96f1c62167d9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:54:31.847169 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.847142    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da3844c-03db-451b-87c0-96f1c62167d9-kube-api-access-8zp9d" pod "8da3844c-03db-451b-87c0-96f1c62167d9" (UID: "8da3844c-03db-451b-87c0-96f1c62167d9"). InnerVolumeSpecName "kube-api-access-8zp9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:54:31.847272 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.847208    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8da3844c-03db-451b-87c0-96f1c62167d9-console-serving-cert" pod "8da3844c-03db-451b-87c0-96f1c62167d9" (UID: "8da3844c-03db-451b-87c0-96f1c62167d9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:54:31.847350 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.847334    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8da3844c-03db-451b-87c0-96f1c62167d9-console-oauth-config" pod "8da3844c-03db-451b-87c0-96f1c62167d9" (UID: "8da3844c-03db-451b-87c0-96f1c62167d9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:54:31.855917 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.855878    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-58b4bd6669-gzcxn"]
Nov 03 07:54:31.856354 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.856335    5070 cpu_manager.go:414] "containerMap: removing container" podUID="119141fd-3c4f-4b94-9f18-6892986867e9" containerName="registry"
Nov 03 07:54:31.856434 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.856357    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119141fd-3c4f-4b94-9f18-6892986867e9" containerName="registry"
Nov 03 07:54:31.856434 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.856385    5070 cpu_manager.go:414] "containerMap: removing container" podUID="5fbc88d3-6002-4ab2-beed-7cd2c55ec205" containerName="console"
Nov 03 07:54:31.856434 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.856393    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fbc88d3-6002-4ab2-beed-7cd2c55ec205" containerName="console"
Nov 03 07:54:31.856434 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.856416    5070 cpu_manager.go:414] "containerMap: removing container" podUID="df604a27-2a06-42a0-bcbd-f7a10c1420c1" containerName="registry"
Nov 03 07:54:31.856434 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.856423    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df604a27-2a06-42a0-bcbd-f7a10c1420c1" containerName="registry"
Nov 03 07:54:31.856434 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.856434    5070 cpu_manager.go:414] "containerMap: removing container" podUID="8da3844c-03db-451b-87c0-96f1c62167d9" containerName="console"
Nov 03 07:54:31.856649 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.856441    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8da3844c-03db-451b-87c0-96f1c62167d9" containerName="console"
Nov 03 07:54:31.856649 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.856452    5070 cpu_manager.go:414] "containerMap: removing container" podUID="66e53c53-b31d-43f7-b7b2-2c90a07dcf09" containerName="insights-gathering"
Nov 03 07:54:31.856649 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.856459    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e53c53-b31d-43f7-b7b2-2c90a07dcf09" containerName="insights-gathering"
Nov 03 07:54:31.856649 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.856556    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fbc88d3-6002-4ab2-beed-7cd2c55ec205" containerName="console"
Nov 03 07:54:31.856649 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.856570    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="df604a27-2a06-42a0-bcbd-f7a10c1420c1" containerName="registry"
Nov 03 07:54:31.856649 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.856578    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="8da3844c-03db-451b-87c0-96f1c62167d9" containerName="console"
Nov 03 07:54:31.856649 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.856588    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e53c53-b31d-43f7-b7b2-2c90a07dcf09" containerName="insights-gathering"
Nov 03 07:54:31.856649 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.856597    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="119141fd-3c4f-4b94-9f18-6892986867e9" containerName="registry"
Nov 03 07:54:31.875915 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.875821    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58b4bd6669-gzcxn"]
Nov 03 07:54:31.876013 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.875953    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:31.946020 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.945981    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/873e56e5-1f64-4eec-a578-bf4ddd072869-console-config\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:31.946270 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.946026    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/873e56e5-1f64-4eec-a578-bf4ddd072869-trusted-ca-bundle\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:31.946270 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.946136    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/873e56e5-1f64-4eec-a578-bf4ddd072869-tmp\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:31.946270 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.946175    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/873e56e5-1f64-4eec-a578-bf4ddd072869-oauth-serving-cert\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:31.946270 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.946201    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2p2g\" (UniqueName: \"kubernetes.io/projected/873e56e5-1f64-4eec-a578-bf4ddd072869-kube-api-access-h2p2g\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:31.946469 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.946290    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/873e56e5-1f64-4eec-a578-bf4ddd072869-service-ca\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:31.946469 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.946345    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/873e56e5-1f64-4eec-a578-bf4ddd072869-console-serving-cert\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:31.946469 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.946402    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/873e56e5-1f64-4eec-a578-bf4ddd072869-console-oauth-config\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:31.946576 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.946491    5070 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8da3844c-03db-451b-87c0-96f1c62167d9-console-oauth-config\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:31.946576 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.946508    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zp9d\" (UniqueName: \"kubernetes.io/projected/8da3844c-03db-451b-87c0-96f1c62167d9-kube-api-access-8zp9d\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:31.946576 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.946522    5070 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8da3844c-03db-451b-87c0-96f1c62167d9-console-serving-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:31.946576 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.946531    5070 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8da3844c-03db-451b-87c0-96f1c62167d9-oauth-serving-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:31.946576 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.946541    5070 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8da3844c-03db-451b-87c0-96f1c62167d9-service-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:31.946576 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:31.946550    5070 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8da3844c-03db-451b-87c0-96f1c62167d9-tmp\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:54:32.047238 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.047214    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/873e56e5-1f64-4eec-a578-bf4ddd072869-console-config\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:32.047312 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.047241    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/873e56e5-1f64-4eec-a578-bf4ddd072869-trusted-ca-bundle\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:32.047312 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.047266    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/873e56e5-1f64-4eec-a578-bf4ddd072869-tmp\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:32.047396 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.047375    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/873e56e5-1f64-4eec-a578-bf4ddd072869-oauth-serving-cert\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:32.047444 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.047418    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2p2g\" (UniqueName: \"kubernetes.io/projected/873e56e5-1f64-4eec-a578-bf4ddd072869-kube-api-access-h2p2g\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:32.047496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.047473    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/873e56e5-1f64-4eec-a578-bf4ddd072869-service-ca\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:32.047549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.047501    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/873e56e5-1f64-4eec-a578-bf4ddd072869-console-serving-cert\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:32.047549 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.047525    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/873e56e5-1f64-4eec-a578-bf4ddd072869-console-oauth-config\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:32.047723 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.047705    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/873e56e5-1f64-4eec-a578-bf4ddd072869-tmp\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:32.048003 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.047981    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/873e56e5-1f64-4eec-a578-bf4ddd072869-console-config\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:32.048174 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.048155    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/873e56e5-1f64-4eec-a578-bf4ddd072869-oauth-serving-cert\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:32.048237 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.048181    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/873e56e5-1f64-4eec-a578-bf4ddd072869-service-ca\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:32.048292 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.048238    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/873e56e5-1f64-4eec-a578-bf4ddd072869-trusted-ca-bundle\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:32.049959 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.049938    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/873e56e5-1f64-4eec-a578-bf4ddd072869-console-serving-cert\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:32.050045 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.049981    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/873e56e5-1f64-4eec-a578-bf4ddd072869-console-oauth-config\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:32.086251 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.086231    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2p2g\" (UniqueName: \"kubernetes.io/projected/873e56e5-1f64-4eec-a578-bf4ddd072869-kube-api-access-h2p2g\") pod \"console-58b4bd6669-gzcxn\" (UID: \"873e56e5-1f64-4eec-a578-bf4ddd072869\") " pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:32.203348 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.203326    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:32.492902 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.492879    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58b4bd6669-gzcxn"]
Nov 03 07:54:32.494849 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:54:32.494823    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873e56e5_1f64_4eec_a578_bf4ddd072869.slice/crio-e4278d77f6458964d706a722ff19a805d0012e5d4150910ce115ec9295381995 WatchSource:0}: Error finding container e4278d77f6458964d706a722ff19a805d0012e5d4150910ce115ec9295381995: Status 404 returned error can't find the container with id e4278d77f6458964d706a722ff19a805d0012e5d4150910ce115ec9295381995
Nov 03 07:54:32.545857 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.545827    5070 generic.go:356] "Generic (PLEG): container finished" podID="8da3844c-03db-451b-87c0-96f1c62167d9" containerID="b82b1f8696ece0e83101481c4adff477ede3670fe8936b0cd1576770fe9352e7" exitCode=0
Nov 03 07:54:32.545973 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.545862    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6df9f67-6dfhh" event={"ID":"8da3844c-03db-451b-87c0-96f1c62167d9","Type":"ContainerDied","Data":"b82b1f8696ece0e83101481c4adff477ede3670fe8936b0cd1576770fe9352e7"}
Nov 03 07:54:32.545973 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.545894    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f6df9f67-6dfhh"
Nov 03 07:54:32.545973 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.545923    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f6df9f67-6dfhh" event={"ID":"8da3844c-03db-451b-87c0-96f1c62167d9","Type":"ContainerDied","Data":"babcb86d729169ca7fc680c1c925439af5d4158fc3e57c26f6b2cc71ee00b2ff"}
Nov 03 07:54:32.545973 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.545954    5070 scope.go:122] "RemoveContainer" containerID="b82b1f8696ece0e83101481c4adff477ede3670fe8936b0cd1576770fe9352e7"
Nov 03 07:54:32.547403 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.547378    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b4bd6669-gzcxn" event={"ID":"873e56e5-1f64-4eec-a578-bf4ddd072869","Type":"ContainerStarted","Data":"e4278d77f6458964d706a722ff19a805d0012e5d4150910ce115ec9295381995"}
Nov 03 07:54:32.553998 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.553975    5070 scope.go:122] "RemoveContainer" containerID="b82b1f8696ece0e83101481c4adff477ede3670fe8936b0cd1576770fe9352e7"
Nov 03 07:54:32.554270 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:54:32.554250    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b82b1f8696ece0e83101481c4adff477ede3670fe8936b0cd1576770fe9352e7\": container with ID starting with b82b1f8696ece0e83101481c4adff477ede3670fe8936b0cd1576770fe9352e7 not found: ID does not exist" containerID="b82b1f8696ece0e83101481c4adff477ede3670fe8936b0cd1576770fe9352e7"
Nov 03 07:54:32.554326 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.554280    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b82b1f8696ece0e83101481c4adff477ede3670fe8936b0cd1576770fe9352e7"} err="failed to get container status \"b82b1f8696ece0e83101481c4adff477ede3670fe8936b0cd1576770fe9352e7\": rpc error: code = NotFound desc = could not find container \"b82b1f8696ece0e83101481c4adff477ede3670fe8936b0cd1576770fe9352e7\": container with ID starting with b82b1f8696ece0e83101481c4adff477ede3670fe8936b0cd1576770fe9352e7 not found: ID does not exist"
Nov 03 07:54:32.558717 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.558698    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f6df9f67-6dfhh"]
Nov 03 07:54:32.562538 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:32.562517    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f6df9f67-6dfhh"]
Nov 03 07:54:33.553751 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:33.553714    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b4bd6669-gzcxn" event={"ID":"873e56e5-1f64-4eec-a578-bf4ddd072869","Type":"ContainerStarted","Data":"30b770dc31a8e221d7e448c5e887fcffafb942ff56edc80f278bb05c7c34fa18"}
Nov 03 07:54:33.568645 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:33.568602    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-58b4bd6669-gzcxn" podStartSLOduration=2.568589703 podStartE2EDuration="2.568589703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:54:33.568444169 +0000 UTC m=+71.682908111" watchObservedRunningTime="2026-11-03 07:54:33.568589703 +0000 UTC m=+71.683053633"
Nov 03 07:54:34.023531 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:34.023501    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8da3844c-03db-451b-87c0-96f1c62167d9" path="/var/lib/kubelet/pods/8da3844c-03db-451b-87c0-96f1c62167d9/volumes"
Nov 03 07:54:42.204127 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:42.204096    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:42.204606 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:42.204155    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:42.208838 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:42.208818    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:42.596919 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:42.596850    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 07:54:42.640449 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:42.640418    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f8b567755-rww99"]
Nov 03 07:54:51.631947 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:51.631899    5070 generic.go:356] "Generic (PLEG): container finished" podID="58448f55-1670-41ce-8e35-182fe94f2d7e" containerID="17f816370e04394a061fee875159364543025fd7ac60dca582b06ada5f984d25" exitCode=0
Nov 03 07:54:51.632330 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:51.631964    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f" event={"ID":"58448f55-1670-41ce-8e35-182fe94f2d7e","Type":"ContainerDied","Data":"17f816370e04394a061fee875159364543025fd7ac60dca582b06ada5f984d25"}
Nov 03 07:54:51.632330 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:51.632261    5070 scope.go:122] "RemoveContainer" containerID="17f816370e04394a061fee875159364543025fd7ac60dca582b06ada5f984d25"
Nov 03 07:54:52.049186 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:54:52.049151    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 07:54:52.637188 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:52.637152    5070 generic.go:356] "Generic (PLEG): container finished" podID="e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3" containerID="b4a05aaeeb5ccdf147f3672d11c6d2a259fea578dd06392020786c013e31fff8" exitCode=0
Nov 03 07:54:52.637587 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:52.637226    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk" event={"ID":"e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3","Type":"ContainerDied","Data":"b4a05aaeeb5ccdf147f3672d11c6d2a259fea578dd06392020786c013e31fff8"}
Nov 03 07:54:52.637587 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:52.637566    5070 scope.go:122] "RemoveContainer" containerID="b4a05aaeeb5ccdf147f3672d11c6d2a259fea578dd06392020786c013e31fff8"
Nov 03 07:54:52.639219 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:52.639199    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-gt59f" event={"ID":"58448f55-1670-41ce-8e35-182fe94f2d7e","Type":"ContainerStarted","Data":"9d2d0127d715cd8e68ade392887964541f83d5391402272303590da02dd7568e"}
Nov 03 07:54:53.644904 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:54:53.644868    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2zhzk" event={"ID":"e3b04a92-4ec8-41b7-a9d4-a559dc80a7c3","Type":"ContainerStarted","Data":"d5ed98ba4bfe34b079ee6aa6416b40a12f2f3dfc255266de05263cddccb8e7b5"}
Nov 03 07:55:02.684909 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:02.684842    5070 generic.go:356] "Generic (PLEG): container finished" podID="2c197040-3bfe-416c-be23-5d5b67678173" containerID="9cb92312aadc9a572f614677df0e2b56a829ba9a81fbce2e8230dc65f5292b8e" exitCode=0
Nov 03 07:55:02.684909 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:02.684897    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kfr2h-k95nf" event={"ID":"2c197040-3bfe-416c-be23-5d5b67678173","Type":"ContainerDied","Data":"9cb92312aadc9a572f614677df0e2b56a829ba9a81fbce2e8230dc65f5292b8e"}
Nov 03 07:55:03.825453 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:03.825429    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kfr2h-k95nf"
Nov 03 07:55:03.984858 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:03.984788    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c197040-3bfe-416c-be23-5d5b67678173-kube-api-access-ks4mg\" (UniqueName: \"kubernetes.io/projected/2c197040-3bfe-416c-be23-5d5b67678173-kube-api-access-ks4mg\") pod \"2c197040-3bfe-416c-be23-5d5b67678173\" (UID: \"2c197040-3bfe-416c-be23-5d5b67678173\") "
Nov 03 07:55:03.984858 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:03.984831    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c197040-3bfe-416c-be23-5d5b67678173-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c197040-3bfe-416c-be23-5d5b67678173-service-ca-bundle\") pod \"2c197040-3bfe-416c-be23-5d5b67678173\" (UID: \"2c197040-3bfe-416c-be23-5d5b67678173\") "
Nov 03 07:55:03.985059 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:03.984878    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c197040-3bfe-416c-be23-5d5b67678173-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c197040-3bfe-416c-be23-5d5b67678173-trusted-ca-bundle\") pod \"2c197040-3bfe-416c-be23-5d5b67678173\" (UID: \"2c197040-3bfe-416c-be23-5d5b67678173\") "
Nov 03 07:55:03.985059 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:03.984904    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c197040-3bfe-416c-be23-5d5b67678173-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2c197040-3bfe-416c-be23-5d5b67678173-archives-path\") pod \"2c197040-3bfe-416c-be23-5d5b67678173\" (UID: \"2c197040-3bfe-416c-be23-5d5b67678173\") "
Nov 03 07:55:03.985271 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:03.985243    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c197040-3bfe-416c-be23-5d5b67678173-trusted-ca-bundle" pod "2c197040-3bfe-416c-be23-5d5b67678173" (UID: "2c197040-3bfe-416c-be23-5d5b67678173"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:55:03.985326 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:03.985249    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c197040-3bfe-416c-be23-5d5b67678173-service-ca-bundle" pod "2c197040-3bfe-416c-be23-5d5b67678173" (UID: "2c197040-3bfe-416c-be23-5d5b67678173"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:55:03.985326 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:03.985256    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c197040-3bfe-416c-be23-5d5b67678173-archives-path" pod "2c197040-3bfe-416c-be23-5d5b67678173" (UID: "2c197040-3bfe-416c-be23-5d5b67678173"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:55:03.986835 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:03.986813    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c197040-3bfe-416c-be23-5d5b67678173-kube-api-access-ks4mg" pod "2c197040-3bfe-416c-be23-5d5b67678173" (UID: "2c197040-3bfe-416c-be23-5d5b67678173"). InnerVolumeSpecName "kube-api-access-ks4mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:55:04.086082 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:04.086059    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks4mg\" (UniqueName: \"kubernetes.io/projected/2c197040-3bfe-416c-be23-5d5b67678173-kube-api-access-ks4mg\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:55:04.086082 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:04.086081    5070 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c197040-3bfe-416c-be23-5d5b67678173-service-ca-bundle\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:55:04.086206 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:04.086091    5070 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2c197040-3bfe-416c-be23-5d5b67678173-trusted-ca-bundle\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:55:04.086206 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:04.086101    5070 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2c197040-3bfe-416c-be23-5d5b67678173-archives-path\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:55:04.694382 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:04.694351    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-kfr2h-k95nf" event={"ID":"2c197040-3bfe-416c-be23-5d5b67678173","Type":"ContainerDied","Data":"02a0585ede1251e50d2abdba9af0b2fba7aaa691d5836c9cd0779d87bb79f69b"}
Nov 03 07:55:04.694382 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:04.694376    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-kfr2h-k95nf"
Nov 03 07:55:04.694562 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:04.694400    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02a0585ede1251e50d2abdba9af0b2fba7aaa691d5836c9cd0779d87bb79f69b"
Nov 03 07:55:07.661752 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:07.661690    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6f8b567755-rww99" podUID="3b395996-50fb-416f-a744-fffab9e943f9" containerName="console" containerID="cri-o://89a74c07fc8ff4cf1d364480e57d9a1c1613f6d15cd42c9ecd4116c1d102f618" gracePeriod=15
Nov 03 07:55:07.913056 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:07.913000    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:55:08.012456 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.012426    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-service-ca\") pod \"3b395996-50fb-416f-a744-fffab9e943f9\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") "
Nov 03 07:55:08.012606 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.012463    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-trusted-ca-bundle\") pod \"3b395996-50fb-416f-a744-fffab9e943f9\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") "
Nov 03 07:55:08.012606 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.012496    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b395996-50fb-416f-a744-fffab9e943f9-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b395996-50fb-416f-a744-fffab9e943f9-tmp\") pod \"3b395996-50fb-416f-a744-fffab9e943f9\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") "
Nov 03 07:55:08.012734 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.012701    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b395996-50fb-416f-a744-fffab9e943f9-kube-api-access-qlx9t\" (UniqueName: \"kubernetes.io/projected/3b395996-50fb-416f-a744-fffab9e943f9-kube-api-access-qlx9t\") pod \"3b395996-50fb-416f-a744-fffab9e943f9\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") "
Nov 03 07:55:08.012823 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.012790    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b395996-50fb-416f-a744-fffab9e943f9-tmp" pod "3b395996-50fb-416f-a744-fffab9e943f9" (UID: "3b395996-50fb-416f-a744-fffab9e943f9"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:55:08.012823 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.012801    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b395996-50fb-416f-a744-fffab9e943f9-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b395996-50fb-416f-a744-fffab9e943f9-console-oauth-config\") pod \"3b395996-50fb-416f-a744-fffab9e943f9\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") "
Nov 03 07:55:08.012823 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.012807    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-service-ca" pod "3b395996-50fb-416f-a744-fffab9e943f9" (UID: "3b395996-50fb-416f-a744-fffab9e943f9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:55:08.012933 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.012851    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-console-config\" (UniqueName: \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-console-config\") pod \"3b395996-50fb-416f-a744-fffab9e943f9\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") "
Nov 03 07:55:08.012933 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.012853    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-trusted-ca-bundle" pod "3b395996-50fb-416f-a744-fffab9e943f9" (UID: "3b395996-50fb-416f-a744-fffab9e943f9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:55:08.012933 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.012890    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-oauth-serving-cert\") pod \"3b395996-50fb-416f-a744-fffab9e943f9\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") "
Nov 03 07:55:08.013057 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.012940    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b395996-50fb-416f-a744-fffab9e943f9-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b395996-50fb-416f-a744-fffab9e943f9-console-serving-cert\") pod \"3b395996-50fb-416f-a744-fffab9e943f9\" (UID: \"3b395996-50fb-416f-a744-fffab9e943f9\") "
Nov 03 07:55:08.013244 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.013214    5070 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-service-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:55:08.013244 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.013240    5070 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-trusted-ca-bundle\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:55:08.013355 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.013241    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-console-config" pod "3b395996-50fb-416f-a744-fffab9e943f9" (UID: "3b395996-50fb-416f-a744-fffab9e943f9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:55:08.013355 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.013254    5070 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b395996-50fb-416f-a744-fffab9e943f9-tmp\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:55:08.013355 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.013273    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-oauth-serving-cert" pod "3b395996-50fb-416f-a744-fffab9e943f9" (UID: "3b395996-50fb-416f-a744-fffab9e943f9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 07:55:08.014805 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.014781    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b395996-50fb-416f-a744-fffab9e943f9-kube-api-access-qlx9t" pod "3b395996-50fb-416f-a744-fffab9e943f9" (UID: "3b395996-50fb-416f-a744-fffab9e943f9"). InnerVolumeSpecName "kube-api-access-qlx9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:55:08.014918 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.014899    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b395996-50fb-416f-a744-fffab9e943f9-console-serving-cert" pod "3b395996-50fb-416f-a744-fffab9e943f9" (UID: "3b395996-50fb-416f-a744-fffab9e943f9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:55:08.014960 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.014918    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b395996-50fb-416f-a744-fffab9e943f9-console-oauth-config" pod "3b395996-50fb-416f-a744-fffab9e943f9" (UID: "3b395996-50fb-416f-a744-fffab9e943f9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 07:55:08.113568 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.113541    5070 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-console-config\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:55:08.113568 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.113564    5070 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b395996-50fb-416f-a744-fffab9e943f9-oauth-serving-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:55:08.113704 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.113573    5070 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b395996-50fb-416f-a744-fffab9e943f9-console-serving-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:55:08.113704 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.113582    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlx9t\" (UniqueName: \"kubernetes.io/projected/3b395996-50fb-416f-a744-fffab9e943f9-kube-api-access-qlx9t\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:55:08.113704 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.113592    5070 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b395996-50fb-416f-a744-fffab9e943f9-console-oauth-config\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:55:08.711378 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.711348    5070 generic.go:356] "Generic (PLEG): container finished" podID="3b395996-50fb-416f-a744-fffab9e943f9" containerID="89a74c07fc8ff4cf1d364480e57d9a1c1613f6d15cd42c9ecd4116c1d102f618" exitCode=0
Nov 03 07:55:08.711770 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.711394    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f8b567755-rww99"
Nov 03 07:55:08.711770 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.711430    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f8b567755-rww99" event={"ID":"3b395996-50fb-416f-a744-fffab9e943f9","Type":"ContainerDied","Data":"89a74c07fc8ff4cf1d364480e57d9a1c1613f6d15cd42c9ecd4116c1d102f618"}
Nov 03 07:55:08.711770 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.711486    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f8b567755-rww99" event={"ID":"3b395996-50fb-416f-a744-fffab9e943f9","Type":"ContainerDied","Data":"6bdba2c53ad4835b4959da433e38bbde62e378039a9be1506d17e7f08605bfeb"}
Nov 03 07:55:08.711770 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.711508    5070 scope.go:122] "RemoveContainer" containerID="89a74c07fc8ff4cf1d364480e57d9a1c1613f6d15cd42c9ecd4116c1d102f618"
Nov 03 07:55:08.719739 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.719717    5070 scope.go:122] "RemoveContainer" containerID="89a74c07fc8ff4cf1d364480e57d9a1c1613f6d15cd42c9ecd4116c1d102f618"
Nov 03 07:55:08.720038 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:55:08.720013    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a74c07fc8ff4cf1d364480e57d9a1c1613f6d15cd42c9ecd4116c1d102f618\": container with ID starting with 89a74c07fc8ff4cf1d364480e57d9a1c1613f6d15cd42c9ecd4116c1d102f618 not found: ID does not exist" containerID="89a74c07fc8ff4cf1d364480e57d9a1c1613f6d15cd42c9ecd4116c1d102f618"
Nov 03 07:55:08.720111 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.720051    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a74c07fc8ff4cf1d364480e57d9a1c1613f6d15cd42c9ecd4116c1d102f618"} err="failed to get container status \"89a74c07fc8ff4cf1d364480e57d9a1c1613f6d15cd42c9ecd4116c1d102f618\": rpc error: code = NotFound desc = could not find container \"89a74c07fc8ff4cf1d364480e57d9a1c1613f6d15cd42c9ecd4116c1d102f618\": container with ID starting with 89a74c07fc8ff4cf1d364480e57d9a1c1613f6d15cd42c9ecd4116c1d102f618 not found: ID does not exist"
Nov 03 07:55:08.725070 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.725048    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f8b567755-rww99"]
Nov 03 07:55:08.730798 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:08.730774    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f8b567755-rww99"]
Nov 03 07:55:10.023697 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:55:10.023656    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b395996-50fb-416f-a744-fffab9e943f9" path="/var/lib/kubelet/pods/3b395996-50fb-416f-a744-fffab9e943f9/volumes"
Nov 03 07:55:22.049791 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:55:22.049747    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 07:55:52.050655 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:55:52.050571    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 07:56:22.053387 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:56:22.053345    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 07:56:52.051099 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:56:52.051061    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 07:57:22.050649 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:57:22.050609    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 07:57:22.497658 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.497621    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw"]
Nov 03 07:57:22.498007 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.497984    5070 cpu_manager.go:414] "containerMap: removing container" podUID="3b395996-50fb-416f-a744-fffab9e943f9" containerName="console"
Nov 03 07:57:22.498007 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.498001    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b395996-50fb-416f-a744-fffab9e943f9" containerName="console"
Nov 03 07:57:22.498280 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.498026    5070 cpu_manager.go:414] "containerMap: removing container" podUID="2c197040-3bfe-416c-be23-5d5b67678173" containerName="insights-gathering"
Nov 03 07:57:22.498280 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.498031    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c197040-3bfe-416c-be23-5d5b67678173" containerName="insights-gathering"
Nov 03 07:57:22.498280 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.498094    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c197040-3bfe-416c-be23-5d5b67678173" containerName="insights-gathering"
Nov 03 07:57:22.498280 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.498104    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b395996-50fb-416f-a744-fffab9e943f9" containerName="console"
Nov 03 07:57:22.500169 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.500150    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw"
Nov 03 07:57:22.502634 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.502608    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 07:57:22.502740 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.502652    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 07:57:22.502740 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.502712    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-v9mbj\""
Nov 03 07:57:22.510150 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.510132    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw"]
Nov 03 07:57:22.524454 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.524427    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk"]
Nov 03 07:57:22.526767 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.526741    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk"
Nov 03 07:57:22.535389 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.535361    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk"]
Nov 03 07:57:22.559196 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.559176    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f5866f9-4bca-42e3-b433-63c4bace7584-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw\" (UID: \"7f5866f9-4bca-42e3-b433-63c4bace7584\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw"
Nov 03 07:57:22.559325 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.559230    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrl7q\" (UniqueName: \"kubernetes.io/projected/7f5866f9-4bca-42e3-b433-63c4bace7584-kube-api-access-wrl7q\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw\" (UID: \"7f5866f9-4bca-42e3-b433-63c4bace7584\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw"
Nov 03 07:57:22.559386 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.559323    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f5866f9-4bca-42e3-b433-63c4bace7584-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw\" (UID: \"7f5866f9-4bca-42e3-b433-63c4bace7584\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw"
Nov 03 07:57:22.559386 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.559354    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/165d70de-22b4-4a6f-8f3e-3eb07968cf6e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk\" (UID: \"165d70de-22b4-4a6f-8f3e-3eb07968cf6e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk"
Nov 03 07:57:22.559386 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.559382    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/165d70de-22b4-4a6f-8f3e-3eb07968cf6e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk\" (UID: \"165d70de-22b4-4a6f-8f3e-3eb07968cf6e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk"
Nov 03 07:57:22.559482 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.559403    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrqvq\" (UniqueName: \"kubernetes.io/projected/165d70de-22b4-4a6f-8f3e-3eb07968cf6e-kube-api-access-wrqvq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk\" (UID: \"165d70de-22b4-4a6f-8f3e-3eb07968cf6e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk"
Nov 03 07:57:22.587444 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.587422    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq"]
Nov 03 07:57:22.589889 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.589875    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq"
Nov 03 07:57:22.598459 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.598440    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq"]
Nov 03 07:57:22.602890 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.602868    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-8hf5l\""
Nov 03 07:57:22.602995 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.602965    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-n6czx\""
Nov 03 07:57:22.659896 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.659868    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9cff832a-3ff3-42f0-98fb-8ba94a053b0a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq\" (UID: \"9cff832a-3ff3-42f0-98fb-8ba94a053b0a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq"
Nov 03 07:57:22.659988 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.659909    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrl7q\" (UniqueName: \"kubernetes.io/projected/7f5866f9-4bca-42e3-b433-63c4bace7584-kube-api-access-wrl7q\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw\" (UID: \"7f5866f9-4bca-42e3-b433-63c4bace7584\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw"
Nov 03 07:57:22.659988 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.659935    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9cff832a-3ff3-42f0-98fb-8ba94a053b0a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq\" (UID: \"9cff832a-3ff3-42f0-98fb-8ba94a053b0a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq"
Nov 03 07:57:22.660114 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.660096    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f5866f9-4bca-42e3-b433-63c4bace7584-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw\" (UID: \"7f5866f9-4bca-42e3-b433-63c4bace7584\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw"
Nov 03 07:57:22.660159 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.660125    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/165d70de-22b4-4a6f-8f3e-3eb07968cf6e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk\" (UID: \"165d70de-22b4-4a6f-8f3e-3eb07968cf6e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk"
Nov 03 07:57:22.660159 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.660143    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/165d70de-22b4-4a6f-8f3e-3eb07968cf6e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk\" (UID: \"165d70de-22b4-4a6f-8f3e-3eb07968cf6e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk"
Nov 03 07:57:22.660222 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.660162    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrqvq\" (UniqueName: \"kubernetes.io/projected/165d70de-22b4-4a6f-8f3e-3eb07968cf6e-kube-api-access-wrqvq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk\" (UID: \"165d70de-22b4-4a6f-8f3e-3eb07968cf6e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk"
Nov 03 07:57:22.660222 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.660198    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f5866f9-4bca-42e3-b433-63c4bace7584-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw\" (UID: \"7f5866f9-4bca-42e3-b433-63c4bace7584\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw"
Nov 03 07:57:22.660295 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.660250    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z46hg\" (UniqueName: \"kubernetes.io/projected/9cff832a-3ff3-42f0-98fb-8ba94a053b0a-kube-api-access-z46hg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq\" (UID: \"9cff832a-3ff3-42f0-98fb-8ba94a053b0a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq"
Nov 03 07:57:22.660529 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.660509    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/165d70de-22b4-4a6f-8f3e-3eb07968cf6e-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk\" (UID: \"165d70de-22b4-4a6f-8f3e-3eb07968cf6e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk"
Nov 03 07:57:22.660529 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.660523    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/165d70de-22b4-4a6f-8f3e-3eb07968cf6e-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk\" (UID: \"165d70de-22b4-4a6f-8f3e-3eb07968cf6e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk"
Nov 03 07:57:22.660622 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.660544    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f5866f9-4bca-42e3-b433-63c4bace7584-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw\" (UID: \"7f5866f9-4bca-42e3-b433-63c4bace7584\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw"
Nov 03 07:57:22.660622 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.660577    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f5866f9-4bca-42e3-b433-63c4bace7584-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw\" (UID: \"7f5866f9-4bca-42e3-b433-63c4bace7584\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw"
Nov 03 07:57:22.667059 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.667040    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrl7q\" (UniqueName: \"kubernetes.io/projected/7f5866f9-4bca-42e3-b433-63c4bace7584-kube-api-access-wrl7q\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw\" (UID: \"7f5866f9-4bca-42e3-b433-63c4bace7584\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw"
Nov 03 07:57:22.667239 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.667217    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrqvq\" (UniqueName: \"kubernetes.io/projected/165d70de-22b4-4a6f-8f3e-3eb07968cf6e-kube-api-access-wrqvq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk\" (UID: \"165d70de-22b4-4a6f-8f3e-3eb07968cf6e\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk"
Nov 03 07:57:22.761133 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.761082    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z46hg\" (UniqueName: \"kubernetes.io/projected/9cff832a-3ff3-42f0-98fb-8ba94a053b0a-kube-api-access-z46hg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq\" (UID: \"9cff832a-3ff3-42f0-98fb-8ba94a053b0a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq"
Nov 03 07:57:22.761133 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.761128    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9cff832a-3ff3-42f0-98fb-8ba94a053b0a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq\" (UID: \"9cff832a-3ff3-42f0-98fb-8ba94a053b0a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq"
Nov 03 07:57:22.761249 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.761164    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9cff832a-3ff3-42f0-98fb-8ba94a053b0a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq\" (UID: \"9cff832a-3ff3-42f0-98fb-8ba94a053b0a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq"
Nov 03 07:57:22.761569 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.761547    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9cff832a-3ff3-42f0-98fb-8ba94a053b0a-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq\" (UID: \"9cff832a-3ff3-42f0-98fb-8ba94a053b0a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq"
Nov 03 07:57:22.761606 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.761568    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9cff832a-3ff3-42f0-98fb-8ba94a053b0a-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq\" (UID: \"9cff832a-3ff3-42f0-98fb-8ba94a053b0a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq"
Nov 03 07:57:22.768974 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.768954    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z46hg\" (UniqueName: \"kubernetes.io/projected/9cff832a-3ff3-42f0-98fb-8ba94a053b0a-kube-api-access-z46hg\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq\" (UID: \"9cff832a-3ff3-42f0-98fb-8ba94a053b0a\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq"
Nov 03 07:57:22.812873 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.812852    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw"
Nov 03 07:57:22.838936 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.838915    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk"
Nov 03 07:57:22.904106 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:22.904008    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq"
Nov 03 07:57:23.082884 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:23.082862    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw"]
Nov 03 07:57:23.084643 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:57:23.084620    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5866f9_4bca_42e3_b433_63c4bace7584.slice/crio-2e1e069332052aa8333dcd14feafdce2afbfb406189179a6e48b85da1f806910 WatchSource:0}: Error finding container 2e1e069332052aa8333dcd14feafdce2afbfb406189179a6e48b85da1f806910: Status 404 returned error can't find the container with id 2e1e069332052aa8333dcd14feafdce2afbfb406189179a6e48b85da1f806910
Nov 03 07:57:23.112397 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:23.112371    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk"]
Nov 03 07:57:23.115161 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:57:23.115139    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165d70de_22b4_4a6f_8f3e_3eb07968cf6e.slice/crio-7c502596dc79629f126a2235ac1aa428bf9b07d5e92d0b2e4b595f0538b7e51f WatchSource:0}: Error finding container 7c502596dc79629f126a2235ac1aa428bf9b07d5e92d0b2e4b595f0538b7e51f: Status 404 returned error can't find the container with id 7c502596dc79629f126a2235ac1aa428bf9b07d5e92d0b2e4b595f0538b7e51f
Nov 03 07:57:23.171500 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:23.171439    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq"]
Nov 03 07:57:23.173336 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:57:23.173314    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cff832a_3ff3_42f0_98fb_8ba94a053b0a.slice/crio-1cebe0f1aa7639a7dbff9b7a89425ffbdfed57ac3580fda34ac3afa475da03b1 WatchSource:0}: Error finding container 1cebe0f1aa7639a7dbff9b7a89425ffbdfed57ac3580fda34ac3afa475da03b1: Status 404 returned error can't find the container with id 1cebe0f1aa7639a7dbff9b7a89425ffbdfed57ac3580fda34ac3afa475da03b1
Nov 03 07:57:23.269509 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:23.269483    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw" event={"ID":"7f5866f9-4bca-42e3-b433-63c4bace7584","Type":"ContainerStarted","Data":"2e1e069332052aa8333dcd14feafdce2afbfb406189179a6e48b85da1f806910"}
Nov 03 07:57:23.270664 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:23.270636    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq" event={"ID":"9cff832a-3ff3-42f0-98fb-8ba94a053b0a","Type":"ContainerStarted","Data":"1cebe0f1aa7639a7dbff9b7a89425ffbdfed57ac3580fda34ac3afa475da03b1"}
Nov 03 07:57:23.271656 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:23.271634    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk" event={"ID":"165d70de-22b4-4a6f-8f3e-3eb07968cf6e","Type":"ContainerStarted","Data":"7c502596dc79629f126a2235ac1aa428bf9b07d5e92d0b2e4b595f0538b7e51f"}
Nov 03 07:57:33.324322 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:33.324239    5070 generic.go:356] "Generic (PLEG): container finished" podID="165d70de-22b4-4a6f-8f3e-3eb07968cf6e" containerID="b6d0c85728e2d140dddf8993c3b2fecdfc8c2679afe472e2a7479dcc3c0f144a" exitCode=0
Nov 03 07:57:33.324776 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:33.324330    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk" event={"ID":"165d70de-22b4-4a6f-8f3e-3eb07968cf6e","Type":"ContainerDied","Data":"b6d0c85728e2d140dddf8993c3b2fecdfc8c2679afe472e2a7479dcc3c0f144a"}
Nov 03 07:57:34.332745 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:34.332714    5070 generic.go:356] "Generic (PLEG): container finished" podID="7f5866f9-4bca-42e3-b433-63c4bace7584" containerID="39d34149d6ea039e805447daa45eb128d0a37bf1814da2723f7e90740931ce93" exitCode=0
Nov 03 07:57:34.333147 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:34.332799    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw" event={"ID":"7f5866f9-4bca-42e3-b433-63c4bace7584","Type":"ContainerDied","Data":"39d34149d6ea039e805447daa45eb128d0a37bf1814da2723f7e90740931ce93"}
Nov 03 07:57:35.339216 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:35.339155    5070 generic.go:356] "Generic (PLEG): container finished" podID="7f5866f9-4bca-42e3-b433-63c4bace7584" containerID="3efcee88e2bb09447c222b1142354f721a5acf87c87ba64c53e0d95d1a9cfc5e" exitCode=0
Nov 03 07:57:35.339562 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:35.339242    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw" event={"ID":"7f5866f9-4bca-42e3-b433-63c4bace7584","Type":"ContainerDied","Data":"3efcee88e2bb09447c222b1142354f721a5acf87c87ba64c53e0d95d1a9cfc5e"}
Nov 03 07:57:35.340961 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:35.340846    5070 generic.go:356] "Generic (PLEG): container finished" podID="9cff832a-3ff3-42f0-98fb-8ba94a053b0a" containerID="e290c6f3cef74ed9367b50192644d6d6b6646a9f18ecc8659e23767cdb76d8e6" exitCode=0
Nov 03 07:57:35.340961 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:35.340920    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq" event={"ID":"9cff832a-3ff3-42f0-98fb-8ba94a053b0a","Type":"ContainerDied","Data":"e290c6f3cef74ed9367b50192644d6d6b6646a9f18ecc8659e23767cdb76d8e6"}
Nov 03 07:57:35.343002 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:35.342981    5070 generic.go:356] "Generic (PLEG): container finished" podID="165d70de-22b4-4a6f-8f3e-3eb07968cf6e" containerID="d05dd106e6a109602d79a5f9b43aa6ed35d6c0477a3d5aea2cdab0d1c3afdbea" exitCode=0
Nov 03 07:57:35.343104 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:35.343057    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk" event={"ID":"165d70de-22b4-4a6f-8f3e-3eb07968cf6e","Type":"ContainerDied","Data":"d05dd106e6a109602d79a5f9b43aa6ed35d6c0477a3d5aea2cdab0d1c3afdbea"}
Nov 03 07:57:36.353145 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:36.352575    5070 generic.go:356] "Generic (PLEG): container finished" podID="9cff832a-3ff3-42f0-98fb-8ba94a053b0a" containerID="f11a899c4563dd38548bb2f35f4f4bfc55d54e149d9c3aefd43d6ebe88cc7b15" exitCode=0
Nov 03 07:57:36.353145 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:36.352660    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq" event={"ID":"9cff832a-3ff3-42f0-98fb-8ba94a053b0a","Type":"ContainerDied","Data":"f11a899c4563dd38548bb2f35f4f4bfc55d54e149d9c3aefd43d6ebe88cc7b15"}
Nov 03 07:57:44.397110 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:44.397067    5070 generic.go:356] "Generic (PLEG): container finished" podID="165d70de-22b4-4a6f-8f3e-3eb07968cf6e" containerID="e644b22a3cbefa9522a877c9ae915bd34e2638a3f8276f545185fac1ec747d39" exitCode=0
Nov 03 07:57:44.397643 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:44.397124    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk" event={"ID":"165d70de-22b4-4a6f-8f3e-3eb07968cf6e","Type":"ContainerDied","Data":"e644b22a3cbefa9522a877c9ae915bd34e2638a3f8276f545185fac1ec747d39"}
Nov 03 07:57:45.404493 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:45.404418    5070 generic.go:356] "Generic (PLEG): container finished" podID="9cff832a-3ff3-42f0-98fb-8ba94a053b0a" containerID="095bfcf116492dae646b35a1b763647059674c0f37ee16e3b4975f3702c37c6a" exitCode=0
Nov 03 07:57:45.404957 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:45.404516    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq" event={"ID":"9cff832a-3ff3-42f0-98fb-8ba94a053b0a","Type":"ContainerDied","Data":"095bfcf116492dae646b35a1b763647059674c0f37ee16e3b4975f3702c37c6a"}
Nov 03 07:57:45.542460 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:45.542437    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk"
Nov 03 07:57:45.676956 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:45.676898    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165d70de-22b4-4a6f-8f3e-3eb07968cf6e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/165d70de-22b4-4a6f-8f3e-3eb07968cf6e-bundle\") pod \"165d70de-22b4-4a6f-8f3e-3eb07968cf6e\" (UID: \"165d70de-22b4-4a6f-8f3e-3eb07968cf6e\") "
Nov 03 07:57:45.676956 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:45.676938    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/165d70de-22b4-4a6f-8f3e-3eb07968cf6e-util\" (UniqueName: \"kubernetes.io/empty-dir/165d70de-22b4-4a6f-8f3e-3eb07968cf6e-util\") pod \"165d70de-22b4-4a6f-8f3e-3eb07968cf6e\" (UID: \"165d70de-22b4-4a6f-8f3e-3eb07968cf6e\") "
Nov 03 07:57:45.677091 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:45.676983    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165d70de-22b4-4a6f-8f3e-3eb07968cf6e-kube-api-access-wrqvq\" (UniqueName: \"kubernetes.io/projected/165d70de-22b4-4a6f-8f3e-3eb07968cf6e-kube-api-access-wrqvq\") pod \"165d70de-22b4-4a6f-8f3e-3eb07968cf6e\" (UID: \"165d70de-22b4-4a6f-8f3e-3eb07968cf6e\") "
Nov 03 07:57:45.677351 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:45.677327    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165d70de-22b4-4a6f-8f3e-3eb07968cf6e-bundle" pod "165d70de-22b4-4a6f-8f3e-3eb07968cf6e" (UID: "165d70de-22b4-4a6f-8f3e-3eb07968cf6e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:57:45.678951 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:45.678929    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165d70de-22b4-4a6f-8f3e-3eb07968cf6e-kube-api-access-wrqvq" pod "165d70de-22b4-4a6f-8f3e-3eb07968cf6e" (UID: "165d70de-22b4-4a6f-8f3e-3eb07968cf6e"). InnerVolumeSpecName "kube-api-access-wrqvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:57:45.681686 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:45.681666    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/165d70de-22b4-4a6f-8f3e-3eb07968cf6e-util" pod "165d70de-22b4-4a6f-8f3e-3eb07968cf6e" (UID: "165d70de-22b4-4a6f-8f3e-3eb07968cf6e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:57:45.778353 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:45.778328    5070 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/165d70de-22b4-4a6f-8f3e-3eb07968cf6e-util\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:57:45.778353 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:45.778352    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrqvq\" (UniqueName: \"kubernetes.io/projected/165d70de-22b4-4a6f-8f3e-3eb07968cf6e-kube-api-access-wrqvq\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:57:45.778496 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:45.778361    5070 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/165d70de-22b4-4a6f-8f3e-3eb07968cf6e-bundle\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:57:46.410155 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:46.410124    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk"
Nov 03 07:57:46.410155 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:46.410141    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d744xwk" event={"ID":"165d70de-22b4-4a6f-8f3e-3eb07968cf6e","Type":"ContainerDied","Data":"7c502596dc79629f126a2235ac1aa428bf9b07d5e92d0b2e4b595f0538b7e51f"}
Nov 03 07:57:46.410631 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:46.410197    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c502596dc79629f126a2235ac1aa428bf9b07d5e92d0b2e4b595f0538b7e51f"
Nov 03 07:57:46.412326 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:46.412299    5070 generic.go:356] "Generic (PLEG): container finished" podID="7f5866f9-4bca-42e3-b433-63c4bace7584" containerID="8527ddc35cadc0c0609a816fce11db43ee0d79d8667556fd20c0353d6301a82f" exitCode=0
Nov 03 07:57:46.412429 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:46.412389    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw" event={"ID":"7f5866f9-4bca-42e3-b433-63c4bace7584","Type":"ContainerDied","Data":"8527ddc35cadc0c0609a816fce11db43ee0d79d8667556fd20c0353d6301a82f"}
Nov 03 07:57:46.543146 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:46.543125    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq"
Nov 03 07:57:46.686641 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:46.686567    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cff832a-3ff3-42f0-98fb-8ba94a053b0a-util\" (UniqueName: \"kubernetes.io/empty-dir/9cff832a-3ff3-42f0-98fb-8ba94a053b0a-util\") pod \"9cff832a-3ff3-42f0-98fb-8ba94a053b0a\" (UID: \"9cff832a-3ff3-42f0-98fb-8ba94a053b0a\") "
Nov 03 07:57:46.686804 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:46.686662    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cff832a-3ff3-42f0-98fb-8ba94a053b0a-kube-api-access-z46hg\" (UniqueName: \"kubernetes.io/projected/9cff832a-3ff3-42f0-98fb-8ba94a053b0a-kube-api-access-z46hg\") pod \"9cff832a-3ff3-42f0-98fb-8ba94a053b0a\" (UID: \"9cff832a-3ff3-42f0-98fb-8ba94a053b0a\") "
Nov 03 07:57:46.686804 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:46.686687    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cff832a-3ff3-42f0-98fb-8ba94a053b0a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9cff832a-3ff3-42f0-98fb-8ba94a053b0a-bundle\") pod \"9cff832a-3ff3-42f0-98fb-8ba94a053b0a\" (UID: \"9cff832a-3ff3-42f0-98fb-8ba94a053b0a\") "
Nov 03 07:57:46.687341 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:46.687316    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cff832a-3ff3-42f0-98fb-8ba94a053b0a-bundle" pod "9cff832a-3ff3-42f0-98fb-8ba94a053b0a" (UID: "9cff832a-3ff3-42f0-98fb-8ba94a053b0a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:57:46.688789 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:46.688748    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cff832a-3ff3-42f0-98fb-8ba94a053b0a-kube-api-access-z46hg" pod "9cff832a-3ff3-42f0-98fb-8ba94a053b0a" (UID: "9cff832a-3ff3-42f0-98fb-8ba94a053b0a"). InnerVolumeSpecName "kube-api-access-z46hg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:57:46.692181 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:46.692159    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cff832a-3ff3-42f0-98fb-8ba94a053b0a-util" pod "9cff832a-3ff3-42f0-98fb-8ba94a053b0a" (UID: "9cff832a-3ff3-42f0-98fb-8ba94a053b0a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:57:46.788240 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:46.788217    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z46hg\" (UniqueName: \"kubernetes.io/projected/9cff832a-3ff3-42f0-98fb-8ba94a053b0a-kube-api-access-z46hg\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:57:46.788240 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:46.788239    5070 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9cff832a-3ff3-42f0-98fb-8ba94a053b0a-bundle\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:57:46.788367 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:46.788249    5070 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9cff832a-3ff3-42f0-98fb-8ba94a053b0a-util\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:57:47.418135 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:47.418108    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq"
Nov 03 07:57:47.418487 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:47.418138    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847azp7gq" event={"ID":"9cff832a-3ff3-42f0-98fb-8ba94a053b0a","Type":"ContainerDied","Data":"1cebe0f1aa7639a7dbff9b7a89425ffbdfed57ac3580fda34ac3afa475da03b1"}
Nov 03 07:57:47.418487 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:47.418183    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cebe0f1aa7639a7dbff9b7a89425ffbdfed57ac3580fda34ac3afa475da03b1"
Nov 03 07:57:47.543947 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:47.543924    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw"
Nov 03 07:57:47.695626 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:47.695555    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5866f9-4bca-42e3-b433-63c4bace7584-kube-api-access-wrl7q\" (UniqueName: \"kubernetes.io/projected/7f5866f9-4bca-42e3-b433-63c4bace7584-kube-api-access-wrl7q\") pod \"7f5866f9-4bca-42e3-b433-63c4bace7584\" (UID: \"7f5866f9-4bca-42e3-b433-63c4bace7584\") "
Nov 03 07:57:47.695626 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:47.695592    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5866f9-4bca-42e3-b433-63c4bace7584-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f5866f9-4bca-42e3-b433-63c4bace7584-bundle\") pod \"7f5866f9-4bca-42e3-b433-63c4bace7584\" (UID: \"7f5866f9-4bca-42e3-b433-63c4bace7584\") "
Nov 03 07:57:47.695853 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:47.695682    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f5866f9-4bca-42e3-b433-63c4bace7584-util\" (UniqueName: \"kubernetes.io/empty-dir/7f5866f9-4bca-42e3-b433-63c4bace7584-util\") pod \"7f5866f9-4bca-42e3-b433-63c4bace7584\" (UID: \"7f5866f9-4bca-42e3-b433-63c4bace7584\") "
Nov 03 07:57:47.696171 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:47.696149    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5866f9-4bca-42e3-b433-63c4bace7584-bundle" pod "7f5866f9-4bca-42e3-b433-63c4bace7584" (UID: "7f5866f9-4bca-42e3-b433-63c4bace7584"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:57:47.697520 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:47.697499    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5866f9-4bca-42e3-b433-63c4bace7584-kube-api-access-wrl7q" pod "7f5866f9-4bca-42e3-b433-63c4bace7584" (UID: "7f5866f9-4bca-42e3-b433-63c4bace7584"). InnerVolumeSpecName "kube-api-access-wrl7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 07:57:47.700952 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:47.700924    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f5866f9-4bca-42e3-b433-63c4bace7584-util" pod "7f5866f9-4bca-42e3-b433-63c4bace7584" (UID: "7f5866f9-4bca-42e3-b433-63c4bace7584"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 07:57:47.796729 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:47.796691    5070 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f5866f9-4bca-42e3-b433-63c4bace7584-util\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:57:47.796729 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:47.796731    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrl7q\" (UniqueName: \"kubernetes.io/projected/7f5866f9-4bca-42e3-b433-63c4bace7584-kube-api-access-wrl7q\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:57:47.796898 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:47.796745    5070 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f5866f9-4bca-42e3-b433-63c4bace7584-bundle\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 07:57:48.424700 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:48.424674    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw"
Nov 03 07:57:48.425106 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:48.424699    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cct6pw" event={"ID":"7f5866f9-4bca-42e3-b433-63c4bace7584","Type":"ContainerDied","Data":"2e1e069332052aa8333dcd14feafdce2afbfb406189179a6e48b85da1f806910"}
Nov 03 07:57:48.425106 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:48.424740    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e1e069332052aa8333dcd14feafdce2afbfb406189179a6e48b85da1f806910"
Nov 03 07:57:52.051641 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:57:52.051609    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 07:57:55.313597 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.313564    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"]
Nov 03 07:57:55.314099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.313931    5070 cpu_manager.go:414] "containerMap: removing container" podUID="165d70de-22b4-4a6f-8f3e-3eb07968cf6e" containerName="extract"
Nov 03 07:57:55.314099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.313953    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165d70de-22b4-4a6f-8f3e-3eb07968cf6e" containerName="extract"
Nov 03 07:57:55.314099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.313974    5070 cpu_manager.go:414] "containerMap: removing container" podUID="7f5866f9-4bca-42e3-b433-63c4bace7584" containerName="util"
Nov 03 07:57:55.314099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.313982    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5866f9-4bca-42e3-b433-63c4bace7584" containerName="util"
Nov 03 07:57:55.314099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.314004    5070 cpu_manager.go:414] "containerMap: removing container" podUID="7f5866f9-4bca-42e3-b433-63c4bace7584" containerName="extract"
Nov 03 07:57:55.314099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.314012    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5866f9-4bca-42e3-b433-63c4bace7584" containerName="extract"
Nov 03 07:57:55.314099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.314029    5070 cpu_manager.go:414] "containerMap: removing container" podUID="9cff832a-3ff3-42f0-98fb-8ba94a053b0a" containerName="pull"
Nov 03 07:57:55.314099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.314036    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cff832a-3ff3-42f0-98fb-8ba94a053b0a" containerName="pull"
Nov 03 07:57:55.314099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.314047    5070 cpu_manager.go:414] "containerMap: removing container" podUID="9cff832a-3ff3-42f0-98fb-8ba94a053b0a" containerName="extract"
Nov 03 07:57:55.314099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.314054    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cff832a-3ff3-42f0-98fb-8ba94a053b0a" containerName="extract"
Nov 03 07:57:55.314099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.314065    5070 cpu_manager.go:414] "containerMap: removing container" podUID="9cff832a-3ff3-42f0-98fb-8ba94a053b0a" containerName="util"
Nov 03 07:57:55.314099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.314072    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cff832a-3ff3-42f0-98fb-8ba94a053b0a" containerName="util"
Nov 03 07:57:55.314099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.314092    5070 cpu_manager.go:414] "containerMap: removing container" podUID="165d70de-22b4-4a6f-8f3e-3eb07968cf6e" containerName="util"
Nov 03 07:57:55.314099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.314099    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165d70de-22b4-4a6f-8f3e-3eb07968cf6e" containerName="util"
Nov 03 07:57:55.314099 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.314107    5070 cpu_manager.go:414] "containerMap: removing container" podUID="165d70de-22b4-4a6f-8f3e-3eb07968cf6e" containerName="pull"
Nov 03 07:57:55.314735 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.314114    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165d70de-22b4-4a6f-8f3e-3eb07968cf6e" containerName="pull"
Nov 03 07:57:55.314735 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.314122    5070 cpu_manager.go:414] "containerMap: removing container" podUID="7f5866f9-4bca-42e3-b433-63c4bace7584" containerName="pull"
Nov 03 07:57:55.314735 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.314128    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5866f9-4bca-42e3-b433-63c4bace7584" containerName="pull"
Nov 03 07:57:55.314735 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.314229    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="165d70de-22b4-4a6f-8f3e-3eb07968cf6e" containerName="extract"
Nov 03 07:57:55.314735 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.314243    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5866f9-4bca-42e3-b433-63c4bace7584" containerName="extract"
Nov 03 07:57:55.314735 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.314253    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cff832a-3ff3-42f0-98fb-8ba94a053b0a" containerName="extract"
Nov 03 07:57:55.344268 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.344245    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:57:55.349346 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.349310    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"]
Nov 03 07:57:55.403501 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.403477    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 07:57:55.403501 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.403498    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 07:57:55.403647 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.403518    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 07:57:55.403647 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.403519    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 07:57:55.403647 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.403557    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-cclgr\""
Nov 03 07:57:55.403647 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.403564    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 07:57:55.450010 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.449990    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ecd3f55-84e0-4321-8d19-900388b6a91f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-qjx65\" (UID: \"9ecd3f55-84e0-4321-8d19-900388b6a91f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:57:55.450113 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.450019    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ecd3f55-84e0-4321-8d19-900388b6a91f-tmp\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-qjx65\" (UID: \"9ecd3f55-84e0-4321-8d19-900388b6a91f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:57:55.450113 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.450050    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ecd3f55-84e0-4321-8d19-900388b6a91f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-qjx65\" (UID: \"9ecd3f55-84e0-4321-8d19-900388b6a91f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:57:55.450220 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.450197    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ecd3f55-84e0-4321-8d19-900388b6a91f-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-qjx65\" (UID: \"9ecd3f55-84e0-4321-8d19-900388b6a91f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:57:55.450260 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.450250    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg7vc\" (UniqueName: \"kubernetes.io/projected/9ecd3f55-84e0-4321-8d19-900388b6a91f-kube-api-access-zg7vc\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-qjx65\" (UID: \"9ecd3f55-84e0-4321-8d19-900388b6a91f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:57:55.450359 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.450341    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9ecd3f55-84e0-4321-8d19-900388b6a91f-manifests\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-qjx65\" (UID: \"9ecd3f55-84e0-4321-8d19-900388b6a91f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:57:55.551282 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.551259    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9ecd3f55-84e0-4321-8d19-900388b6a91f-manifests\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-qjx65\" (UID: \"9ecd3f55-84e0-4321-8d19-900388b6a91f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:57:55.551379 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.551291    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ecd3f55-84e0-4321-8d19-900388b6a91f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-qjx65\" (UID: \"9ecd3f55-84e0-4321-8d19-900388b6a91f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:57:55.551379 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.551318    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ecd3f55-84e0-4321-8d19-900388b6a91f-tmp\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-qjx65\" (UID: \"9ecd3f55-84e0-4321-8d19-900388b6a91f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:57:55.551463 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.551427    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ecd3f55-84e0-4321-8d19-900388b6a91f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-qjx65\" (UID: \"9ecd3f55-84e0-4321-8d19-900388b6a91f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:57:55.551514 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.551474    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ecd3f55-84e0-4321-8d19-900388b6a91f-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-qjx65\" (UID: \"9ecd3f55-84e0-4321-8d19-900388b6a91f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:57:55.551596 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.551576    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg7vc\" (UniqueName: \"kubernetes.io/projected/9ecd3f55-84e0-4321-8d19-900388b6a91f-kube-api-access-zg7vc\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-qjx65\" (UID: \"9ecd3f55-84e0-4321-8d19-900388b6a91f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:57:55.551715 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.551692    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9ecd3f55-84e0-4321-8d19-900388b6a91f-manifests\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-qjx65\" (UID: \"9ecd3f55-84e0-4321-8d19-900388b6a91f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:57:55.553725 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.553694    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ecd3f55-84e0-4321-8d19-900388b6a91f-tmp\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-qjx65\" (UID: \"9ecd3f55-84e0-4321-8d19-900388b6a91f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:57:55.553852 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.553826    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9ecd3f55-84e0-4321-8d19-900388b6a91f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-qjx65\" (UID: \"9ecd3f55-84e0-4321-8d19-900388b6a91f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:57:55.553967 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.553951    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9ecd3f55-84e0-4321-8d19-900388b6a91f-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-qjx65\" (UID: \"9ecd3f55-84e0-4321-8d19-900388b6a91f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:57:55.554008 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.553971    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9ecd3f55-84e0-4321-8d19-900388b6a91f-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-qjx65\" (UID: \"9ecd3f55-84e0-4321-8d19-900388b6a91f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:57:55.562638 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.562618    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg7vc\" (UniqueName: \"kubernetes.io/projected/9ecd3f55-84e0-4321-8d19-900388b6a91f-kube-api-access-zg7vc\") pod \"opendatahub-operator-controller-manager-6dd9d888fc-qjx65\" (UID: \"9ecd3f55-84e0-4321-8d19-900388b6a91f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:57:55.691048 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.691022    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:57:55.990153 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:55.990123    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"]
Nov 03 07:57:55.991782 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:57:55.991744    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ecd3f55_84e0_4321_8d19_900388b6a91f.slice/crio-609bdaa148bcf2ba4a9d6de12b96a5ab89b18769456e7c2f02504edc4f58bfbc WatchSource:0}: Error finding container 609bdaa148bcf2ba4a9d6de12b96a5ab89b18769456e7c2f02504edc4f58bfbc: Status 404 returned error can't find the container with id 609bdaa148bcf2ba4a9d6de12b96a5ab89b18769456e7c2f02504edc4f58bfbc
Nov 03 07:57:56.460466 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:56.460428    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65" event={"ID":"9ecd3f55-84e0-4321-8d19-900388b6a91f","Type":"ContainerStarted","Data":"609bdaa148bcf2ba4a9d6de12b96a5ab89b18769456e7c2f02504edc4f58bfbc"}
Nov 03 07:57:59.477385 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:59.477350    5070 generic.go:356] "Generic (PLEG): container finished" podID="9ecd3f55-84e0-4321-8d19-900388b6a91f" containerID="5296ea4f98a9e0d23ed0d424ba8d6c16d7f70126339208928c653d4ed1298379" exitCode=0
Nov 03 07:57:59.477711 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:57:59.477408    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65" event={"ID":"9ecd3f55-84e0-4321-8d19-900388b6a91f","Type":"ContainerDied","Data":"5296ea4f98a9e0d23ed0d424ba8d6c16d7f70126339208928c653d4ed1298379"}
Nov 03 07:58:00.485248 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:00.485205    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65" event={"ID":"9ecd3f55-84e0-4321-8d19-900388b6a91f","Type":"ContainerStarted","Data":"da923a33ad03f3e4cbb7235fcb0dbe82c426d767fa9a0045b6aacb3a97ac314a"}
Nov 03 07:58:00.485828 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:00.485326    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:58:00.523580 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:00.523524    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65" podStartSLOduration=1.761029555 podStartE2EDuration="5.523510244s" totalImagesPullingTime="3.29135486s" totalInitContainerRuntime="471.125829ms" isStatefulPod=false podCreationTimestamp="2026-11-03 07:57:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:00.521293894 +0000 UTC m=+278.635757825" watchObservedRunningTime="2026-11-03 07:58:00.523510244 +0000 UTC m=+278.637974175"
Nov 03 07:58:11.492075 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:11.492044    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 07:58:14.262406 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:14.262372    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-8nv9t"]
Nov 03 07:58:14.267859 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:14.267835    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8nv9t"
Nov 03 07:58:14.275392 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:14.275366    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 07:58:14.275536 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:14.275521    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 07:58:14.276318 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:14.276293    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-8nv9t"]
Nov 03 07:58:14.299167 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:14.299134    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01360546-977d-4030-b9b7-eaac4f95fb55-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-8nv9t\" (UID: \"01360546-977d-4030-b9b7-eaac4f95fb55\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8nv9t"
Nov 03 07:58:14.299287 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:14.299192    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zq4z\" (UniqueName: \"kubernetes.io/projected/01360546-977d-4030-b9b7-eaac4f95fb55-kube-api-access-9zq4z\") pod \"cert-manager-cainjector-5c57bbd4fc-8nv9t\" (UID: \"01360546-977d-4030-b9b7-eaac4f95fb55\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8nv9t"
Nov 03 07:58:14.304236 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:14.304219    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-glr9w\""
Nov 03 07:58:14.399669 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:14.399643    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zq4z\" (UniqueName: \"kubernetes.io/projected/01360546-977d-4030-b9b7-eaac4f95fb55-kube-api-access-9zq4z\") pod \"cert-manager-cainjector-5c57bbd4fc-8nv9t\" (UID: \"01360546-977d-4030-b9b7-eaac4f95fb55\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8nv9t"
Nov 03 07:58:14.399796 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:14.399703    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01360546-977d-4030-b9b7-eaac4f95fb55-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-8nv9t\" (UID: \"01360546-977d-4030-b9b7-eaac4f95fb55\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8nv9t"
Nov 03 07:58:14.407611 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:14.407582    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/01360546-977d-4030-b9b7-eaac4f95fb55-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-8nv9t\" (UID: \"01360546-977d-4030-b9b7-eaac4f95fb55\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8nv9t"
Nov 03 07:58:14.407836 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:14.407816    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zq4z\" (UniqueName: \"kubernetes.io/projected/01360546-977d-4030-b9b7-eaac4f95fb55-kube-api-access-9zq4z\") pod \"cert-manager-cainjector-5c57bbd4fc-8nv9t\" (UID: \"01360546-977d-4030-b9b7-eaac4f95fb55\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8nv9t"
Nov 03 07:58:14.588223 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:14.588168    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8nv9t"
Nov 03 07:58:14.862133 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:14.862101    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-8nv9t"]
Nov 03 07:58:14.865654 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:58:14.865626    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01360546_977d_4030_b9b7_eaac4f95fb55.slice/crio-3eb0365c2a21c264ba3025e30559948aa4cbbe624d672a948fbeff09fcd84309 WatchSource:0}: Error finding container 3eb0365c2a21c264ba3025e30559948aa4cbbe624d672a948fbeff09fcd84309: Status 404 returned error can't find the container with id 3eb0365c2a21c264ba3025e30559948aa4cbbe624d672a948fbeff09fcd84309
Nov 03 07:58:15.551010 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:15.550978    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8nv9t" event={"ID":"01360546-977d-4030-b9b7-eaac4f95fb55","Type":"ContainerStarted","Data":"3eb0365c2a21c264ba3025e30559948aa4cbbe624d672a948fbeff09fcd84309"}
Nov 03 07:58:19.572576 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:19.572529    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8nv9t" event={"ID":"01360546-977d-4030-b9b7-eaac4f95fb55","Type":"ContainerStarted","Data":"61855d1bc47310c388095fa7ad45af21c6068b68f905da443921119f8f40987a"}
Nov 03 07:58:19.588043 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:19.587988    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-8nv9t" podStartSLOduration=1.43341251 podStartE2EDuration="5.587974085s" totalImagesPullingTime="4.154561575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:58:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:19.586701705 +0000 UTC m=+297.701165637" watchObservedRunningTime="2026-11-03 07:58:19.587974085 +0000 UTC m=+297.702438013"
Nov 03 07:58:22.038958 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:22.038931    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/2.log"
Nov 03 07:58:22.039526 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:22.039505    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/2.log"
Nov 03 07:58:22.057445 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:58:22.057412    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 07:58:30.153193 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:30.153161    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb"]
Nov 03 07:58:30.158182 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:30.158161    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb"
Nov 03 07:58:30.165077 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:30.164859    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 07:58:30.165077 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:30.165042    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-dkd8v\""
Nov 03 07:58:30.165268 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:30.165186    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 07:58:30.167239 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:30.167217    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb"]
Nov 03 07:58:30.220948 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:30.220923    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjrfv\" (UniqueName: \"kubernetes.io/projected/17267346-b1b3-475b-9d89-1fda6e36aab5-kube-api-access-mjrfv\") pod \"authorino-operator-7b7bb4bc77-6kbrb\" (UID: \"17267346-b1b3-475b-9d89-1fda6e36aab5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb"
Nov 03 07:58:30.321734 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:30.321711    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjrfv\" (UniqueName: \"kubernetes.io/projected/17267346-b1b3-475b-9d89-1fda6e36aab5-kube-api-access-mjrfv\") pod \"authorino-operator-7b7bb4bc77-6kbrb\" (UID: \"17267346-b1b3-475b-9d89-1fda6e36aab5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb"
Nov 03 07:58:30.328489 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:30.328468    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjrfv\" (UniqueName: \"kubernetes.io/projected/17267346-b1b3-475b-9d89-1fda6e36aab5-kube-api-access-mjrfv\") pod \"authorino-operator-7b7bb4bc77-6kbrb\" (UID: \"17267346-b1b3-475b-9d89-1fda6e36aab5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb"
Nov 03 07:58:30.472480 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:30.472428    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb"
Nov 03 07:58:30.750654 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:30.750625    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb"]
Nov 03 07:58:30.753232 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:58:30.753205    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17267346_b1b3_475b_9d89_1fda6e36aab5.slice/crio-9e1a808874ce63a758d249f085ab9bd7a34d9502d29f0d3dad9540c933e34847 WatchSource:0}: Error finding container 9e1a808874ce63a758d249f085ab9bd7a34d9502d29f0d3dad9540c933e34847: Status 404 returned error can't find the container with id 9e1a808874ce63a758d249f085ab9bd7a34d9502d29f0d3dad9540c933e34847
Nov 03 07:58:30.755881 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:30.755867    5070 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 07:58:31.632529 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:31.632497    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb" event={"ID":"17267346-b1b3-475b-9d89-1fda6e36aab5","Type":"ContainerStarted","Data":"9e1a808874ce63a758d249f085ab9bd7a34d9502d29f0d3dad9540c933e34847"}
Nov 03 07:58:31.865230 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:31.865201    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6cf769c4c7-l2kwc"]
Nov 03 07:58:31.873121 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:31.873094    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-l2kwc"
Nov 03 07:58:31.883084 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:31.883025    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6cf769c4c7-l2kwc"]
Nov 03 07:58:31.884997 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:31.884966    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 07:58:31.885142 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:31.885079    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 07:58:31.885142 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:31.885140    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gzpwd\""
Nov 03 07:58:31.943155 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:31.943128    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f-tls-certs\") pod \"kube-auth-proxy-6cf769c4c7-l2kwc\" (UID: \"0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f\") " pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-l2kwc"
Nov 03 07:58:31.943287 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:31.943196    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f-tmp\") pod \"kube-auth-proxy-6cf769c4c7-l2kwc\" (UID: \"0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f\") " pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-l2kwc"
Nov 03 07:58:31.943287 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:31.943275    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqdfj\" (UniqueName: \"kubernetes.io/projected/0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f-kube-api-access-jqdfj\") pod \"kube-auth-proxy-6cf769c4c7-l2kwc\" (UID: \"0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f\") " pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-l2kwc"
Nov 03 07:58:32.044517 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:32.044487    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f-tls-certs\") pod \"kube-auth-proxy-6cf769c4c7-l2kwc\" (UID: \"0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f\") " pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-l2kwc"
Nov 03 07:58:32.044673 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:32.044568    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f-tmp\") pod \"kube-auth-proxy-6cf769c4c7-l2kwc\" (UID: \"0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f\") " pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-l2kwc"
Nov 03 07:58:32.044673 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:32.044592    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqdfj\" (UniqueName: \"kubernetes.io/projected/0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f-kube-api-access-jqdfj\") pod \"kube-auth-proxy-6cf769c4c7-l2kwc\" (UID: \"0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f\") " pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-l2kwc"
Nov 03 07:58:32.046897 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:32.046868    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f-tmp\") pod \"kube-auth-proxy-6cf769c4c7-l2kwc\" (UID: \"0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f\") " pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-l2kwc"
Nov 03 07:58:32.047279 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:32.047253    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f-tls-certs\") pod \"kube-auth-proxy-6cf769c4c7-l2kwc\" (UID: \"0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f\") " pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-l2kwc"
Nov 03 07:58:32.058513 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:32.058485    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqdfj\" (UniqueName: \"kubernetes.io/projected/0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f-kube-api-access-jqdfj\") pod \"kube-auth-proxy-6cf769c4c7-l2kwc\" (UID: \"0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f\") " pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-l2kwc"
Nov 03 07:58:32.190451 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:32.190366    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-l2kwc"
Nov 03 07:58:32.509239 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:32.509159    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6cf769c4c7-l2kwc"]
Nov 03 07:58:32.513649 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:58:32.513622    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9fa25a_f905_4b0e_aa09_24c2a0a1cf8f.slice/crio-9b0ccfc0e6963f74063436191157da1ec0d9d30b3970a4ae417fbe86b1bd9a71 WatchSource:0}: Error finding container 9b0ccfc0e6963f74063436191157da1ec0d9d30b3970a4ae417fbe86b1bd9a71: Status 404 returned error can't find the container with id 9b0ccfc0e6963f74063436191157da1ec0d9d30b3970a4ae417fbe86b1bd9a71
Nov 03 07:58:32.639818 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:32.639784    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-l2kwc" event={"ID":"0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f","Type":"ContainerStarted","Data":"9b0ccfc0e6963f74063436191157da1ec0d9d30b3970a4ae417fbe86b1bd9a71"}
Nov 03 07:58:32.761712 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:32.761640    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-tfqr9"]
Nov 03 07:58:32.767291 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:32.767271    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tfqr9"
Nov 03 07:58:32.774622 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:32.774597    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tfqr9"]
Nov 03 07:58:32.796294 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:32.796269    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-st862\""
Nov 03 07:58:32.852450 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:32.852410    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfn27\" (UniqueName: \"kubernetes.io/projected/61277e6f-ccc4-4d25-a7ca-3dae456b0cdf-kube-api-access-pfn27\") pod \"cert-manager-597b95cb84-tfqr9\" (UID: \"61277e6f-ccc4-4d25-a7ca-3dae456b0cdf\") " pod="cert-manager/cert-manager-597b95cb84-tfqr9"
Nov 03 07:58:32.852577 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:32.852557    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/61277e6f-ccc4-4d25-a7ca-3dae456b0cdf-bound-sa-token\") pod \"cert-manager-597b95cb84-tfqr9\" (UID: \"61277e6f-ccc4-4d25-a7ca-3dae456b0cdf\") " pod="cert-manager/cert-manager-597b95cb84-tfqr9"
Nov 03 07:58:32.953337 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:32.953109    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/61277e6f-ccc4-4d25-a7ca-3dae456b0cdf-bound-sa-token\") pod \"cert-manager-597b95cb84-tfqr9\" (UID: \"61277e6f-ccc4-4d25-a7ca-3dae456b0cdf\") " pod="cert-manager/cert-manager-597b95cb84-tfqr9"
Nov 03 07:58:32.953337 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:32.953196    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfn27\" (UniqueName: \"kubernetes.io/projected/61277e6f-ccc4-4d25-a7ca-3dae456b0cdf-kube-api-access-pfn27\") pod \"cert-manager-597b95cb84-tfqr9\" (UID: \"61277e6f-ccc4-4d25-a7ca-3dae456b0cdf\") " pod="cert-manager/cert-manager-597b95cb84-tfqr9"
Nov 03 07:58:32.965093 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:32.965063    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/61277e6f-ccc4-4d25-a7ca-3dae456b0cdf-bound-sa-token\") pod \"cert-manager-597b95cb84-tfqr9\" (UID: \"61277e6f-ccc4-4d25-a7ca-3dae456b0cdf\") " pod="cert-manager/cert-manager-597b95cb84-tfqr9"
Nov 03 07:58:32.965225 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:32.965138    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfn27\" (UniqueName: \"kubernetes.io/projected/61277e6f-ccc4-4d25-a7ca-3dae456b0cdf-kube-api-access-pfn27\") pod \"cert-manager-597b95cb84-tfqr9\" (UID: \"61277e6f-ccc4-4d25-a7ca-3dae456b0cdf\") " pod="cert-manager/cert-manager-597b95cb84-tfqr9"
Nov 03 07:58:33.080327 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:33.080257    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tfqr9"
Nov 03 07:58:33.365918 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:33.365793    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tfqr9"]
Nov 03 07:58:33.368621 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:58:33.368589    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61277e6f_ccc4_4d25_a7ca_3dae456b0cdf.slice/crio-b87bfbb69dd4ae6a66abe30db36792a6c3e75f52bde699a0019c01532a5e4a4e WatchSource:0}: Error finding container b87bfbb69dd4ae6a66abe30db36792a6c3e75f52bde699a0019c01532a5e4a4e: Status 404 returned error can't find the container with id b87bfbb69dd4ae6a66abe30db36792a6c3e75f52bde699a0019c01532a5e4a4e
Nov 03 07:58:33.648342 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:33.648242    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tfqr9" event={"ID":"61277e6f-ccc4-4d25-a7ca-3dae456b0cdf","Type":"ContainerStarted","Data":"574db1fdb36a303d1630478fa5104288cbebc965289d2dc4ac671925bdf2cf5f"}
Nov 03 07:58:33.648342 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:33.648298    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tfqr9" event={"ID":"61277e6f-ccc4-4d25-a7ca-3dae456b0cdf","Type":"ContainerStarted","Data":"b87bfbb69dd4ae6a66abe30db36792a6c3e75f52bde699a0019c01532a5e4a4e"}
Nov 03 07:58:33.670848 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:33.670796    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-tfqr9" podStartSLOduration=1.670781747 podStartE2EDuration="1.670781747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:33.669082687 +0000 UTC m=+311.783546620" watchObservedRunningTime="2026-11-03 07:58:33.670781747 +0000 UTC m=+311.785245677"
Nov 03 07:58:35.660126 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:35.660090    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb" event={"ID":"17267346-b1b3-475b-9d89-1fda6e36aab5","Type":"ContainerStarted","Data":"433fb34ce9e3ab80ce21604126a5f9e2b68325fb9f3a1f7aeb7694fbc95ec2db"}
Nov 03 07:58:35.660536 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:35.660221    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb"
Nov 03 07:58:35.682174 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:35.682117    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb" podStartSLOduration=1.8134388879999999 podStartE2EDuration="5.682099722s" totalImagesPullingTime="3.868660834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:35.680402539 +0000 UTC m=+313.794866470" watchObservedRunningTime="2026-11-03 07:58:35.682099722 +0000 UTC m=+313.796563654"
Nov 03 07:58:36.667122 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:36.667088    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-l2kwc" event={"ID":"0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f","Type":"ContainerStarted","Data":"3e7f0e9bfcc4bda711c15923b71f890ea1297cb0cf139cfadffcd551fe98245f"}
Nov 03 07:58:36.682410 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:36.682361    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6cf769c4c7-l2kwc" podStartSLOduration=2.030564095 podStartE2EDuration="5.682346322s" totalImagesPullingTime="3.651782227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:58:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:36.680154367 +0000 UTC m=+314.794618298" watchObservedRunningTime="2026-11-03 07:58:36.682346322 +0000 UTC m=+314.796810253"
Nov 03 07:58:46.669173 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:46.669146    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb"
Nov 03 07:58:52.050285 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:58:52.050249    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 07:58:53.707697 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:53.707664    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 07:58:53.711970 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:53.711943    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"
Nov 03 07:58:53.734795 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:53.724264    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 07:58:53.734795 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:53.724527    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qv8vc\""
Nov 03 07:58:53.734795 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:53.729238    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 07:58:53.763910 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:53.763882    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 07:58:53.817392 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:53.817364    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d888577-bc72-4581-a858-b43bd5d7290c-config-file\") pod \"limitador-limitador-7c9b87d574-qp9pc\" (UID: \"7d888577-bc72-4581-a858-b43bd5d7290c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"
Nov 03 07:58:53.817517 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:53.817416    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrtg8\" (UniqueName: \"kubernetes.io/projected/7d888577-bc72-4581-a858-b43bd5d7290c-kube-api-access-mrtg8\") pod \"limitador-limitador-7c9b87d574-qp9pc\" (UID: \"7d888577-bc72-4581-a858-b43bd5d7290c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"
Nov 03 07:58:53.918680 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:53.918646    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrtg8\" (UniqueName: \"kubernetes.io/projected/7d888577-bc72-4581-a858-b43bd5d7290c-kube-api-access-mrtg8\") pod \"limitador-limitador-7c9b87d574-qp9pc\" (UID: \"7d888577-bc72-4581-a858-b43bd5d7290c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"
Nov 03 07:58:53.918844 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:53.918780    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d888577-bc72-4581-a858-b43bd5d7290c-config-file\") pod \"limitador-limitador-7c9b87d574-qp9pc\" (UID: \"7d888577-bc72-4581-a858-b43bd5d7290c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"
Nov 03 07:58:53.919402 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:53.919385    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d888577-bc72-4581-a858-b43bd5d7290c-config-file\") pod \"limitador-limitador-7c9b87d574-qp9pc\" (UID: \"7d888577-bc72-4581-a858-b43bd5d7290c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"
Nov 03 07:58:53.931489 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:53.931466    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrtg8\" (UniqueName: \"kubernetes.io/projected/7d888577-bc72-4581-a858-b43bd5d7290c-kube-api-access-mrtg8\") pod \"limitador-limitador-7c9b87d574-qp9pc\" (UID: \"7d888577-bc72-4581-a858-b43bd5d7290c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"
Nov 03 07:58:54.032349 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:54.032293    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"
Nov 03 07:58:54.305349 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:54.305317    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 07:58:54.307427 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:58:54.307402    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d888577_bc72_4581_a858_b43bd5d7290c.slice/crio-2c9e5ddc720442d738d29bab4953a3d04622bc58cb4308983253ae47069b5a5b WatchSource:0}: Error finding container 2c9e5ddc720442d738d29bab4953a3d04622bc58cb4308983253ae47069b5a5b: Status 404 returned error can't find the container with id 2c9e5ddc720442d738d29bab4953a3d04622bc58cb4308983253ae47069b5a5b
Nov 03 07:58:54.347650 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:54.347625    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-n2bh7"]
Nov 03 07:58:54.350497 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:54.350480    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-n2bh7"
Nov 03 07:58:54.358062 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:54.358035    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-n2bh7"]
Nov 03 07:58:54.358917 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:54.358882    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6k6d7\""
Nov 03 07:58:54.422487 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:54.422461    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4sqw\" (UniqueName: \"kubernetes.io/projected/f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae-kube-api-access-l4sqw\") pod \"authorino-56c6c9ddd5-n2bh7\" (UID: \"f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae\") " pod="kuadrant-system/authorino-56c6c9ddd5-n2bh7"
Nov 03 07:58:54.522985 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:54.522960    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4sqw\" (UniqueName: \"kubernetes.io/projected/f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae-kube-api-access-l4sqw\") pod \"authorino-56c6c9ddd5-n2bh7\" (UID: \"f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae\") " pod="kuadrant-system/authorino-56c6c9ddd5-n2bh7"
Nov 03 07:58:54.530038 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:54.530016    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4sqw\" (UniqueName: \"kubernetes.io/projected/f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae-kube-api-access-l4sqw\") pod \"authorino-56c6c9ddd5-n2bh7\" (UID: \"f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae\") " pod="kuadrant-system/authorino-56c6c9ddd5-n2bh7"
Nov 03 07:58:54.662775 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:54.662730    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-n2bh7"
Nov 03 07:58:54.753467 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:54.753439    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-qp9pc" event={"ID":"7d888577-bc72-4581-a858-b43bd5d7290c","Type":"ContainerStarted","Data":"2c9e5ddc720442d738d29bab4953a3d04622bc58cb4308983253ae47069b5a5b"}
Nov 03 07:58:54.929209 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:54.929187    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-n2bh7"]
Nov 03 07:58:54.931801 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:58:54.931772    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf66e6ff0_2de1_44a1_ae9d_52e5b1a40dae.slice/crio-10ac802000db080e98a3877b37fcc015f69560eed4c13530247050931f9e165b WatchSource:0}: Error finding container 10ac802000db080e98a3877b37fcc015f69560eed4c13530247050931f9e165b: Status 404 returned error can't find the container with id 10ac802000db080e98a3877b37fcc015f69560eed4c13530247050931f9e165b
Nov 03 07:58:55.759862 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:55.759824    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-n2bh7" event={"ID":"f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae","Type":"ContainerStarted","Data":"10ac802000db080e98a3877b37fcc015f69560eed4c13530247050931f9e165b"}
Nov 03 07:58:56.767087 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:56.767051    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-qp9pc" event={"ID":"7d888577-bc72-4581-a858-b43bd5d7290c","Type":"ContainerStarted","Data":"3394c6e866ff5f8dc7721e44cb8e62307ce39b22d6a45fcf20f5125f89a772d4"}
Nov 03 07:58:56.767530 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:56.767177    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"
Nov 03 07:58:56.782404 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:56.782352    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-qp9pc" podStartSLOduration=1.991850576 podStartE2EDuration="3.782335997s" totalImagesPullingTime="1.790485421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:56.780747589 +0000 UTC m=+334.895211520" watchObservedRunningTime="2026-11-03 07:58:56.782335997 +0000 UTC m=+334.896799932"
Nov 03 07:58:58.778882 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:58.778841    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-n2bh7" event={"ID":"f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae","Type":"ContainerStarted","Data":"a110c1cb4b42ec63bdc26ed74f857072e1b3059dbb6d7520b79cdc600e031b47"}
Nov 03 07:58:58.791332 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:58:58.791285    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-n2bh7" podStartSLOduration=1.951160284 podStartE2EDuration="4.791274077s" totalImagesPullingTime="2.840113793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:58:58.790953953 +0000 UTC m=+336.905417882" watchObservedRunningTime="2026-11-03 07:58:58.791274077 +0000 UTC m=+336.905738007"
Nov 03 07:59:07.773038 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:59:07.773005    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"
Nov 03 07:59:22.049297 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:59:22.049263    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 07:59:27.595094 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:59:27.595054    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-mdwfw"]
Nov 03 07:59:27.605928 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:59:27.605899    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw"
Nov 03 07:59:27.611118 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:59:27.611091    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-mdwfw"]
Nov 03 07:59:27.615418 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:59:27.615397    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 07:59:27.615569 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:59:27.615439    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 07:59:27.676344 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:59:27.676317    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d26093c-a60a-4cd8-9d43-8ecded62d895-tls-certs\") pod \"model-serving-api-75bd89fdfc-mdwfw\" (UID: \"0d26093c-a60a-4cd8-9d43-8ecded62d895\") " pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw"
Nov 03 07:59:27.676487 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:59:27.676370    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xqnt\" (UniqueName: \"kubernetes.io/projected/0d26093c-a60a-4cd8-9d43-8ecded62d895-kube-api-access-2xqnt\") pod \"model-serving-api-75bd89fdfc-mdwfw\" (UID: \"0d26093c-a60a-4cd8-9d43-8ecded62d895\") " pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw"
Nov 03 07:59:27.682946 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:59:27.682918    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 03 07:59:27.683078 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:59:27.682985    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-2fshp\""
Nov 03 07:59:27.781186 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:59:27.781144    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xqnt\" (UniqueName: \"kubernetes.io/projected/0d26093c-a60a-4cd8-9d43-8ecded62d895-kube-api-access-2xqnt\") pod \"model-serving-api-75bd89fdfc-mdwfw\" (UID: \"0d26093c-a60a-4cd8-9d43-8ecded62d895\") " pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw"
Nov 03 07:59:27.781421 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:59:27.781253    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d26093c-a60a-4cd8-9d43-8ecded62d895-tls-certs\") pod \"model-serving-api-75bd89fdfc-mdwfw\" (UID: \"0d26093c-a60a-4cd8-9d43-8ecded62d895\") " pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw"
Nov 03 07:59:27.786724 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:59:27.786697    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d26093c-a60a-4cd8-9d43-8ecded62d895-tls-certs\") pod \"model-serving-api-75bd89fdfc-mdwfw\" (UID: \"0d26093c-a60a-4cd8-9d43-8ecded62d895\") " pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw"
Nov 03 07:59:27.791065 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:59:27.791037    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xqnt\" (UniqueName: \"kubernetes.io/projected/0d26093c-a60a-4cd8-9d43-8ecded62d895-kube-api-access-2xqnt\") pod \"model-serving-api-75bd89fdfc-mdwfw\" (UID: \"0d26093c-a60a-4cd8-9d43-8ecded62d895\") " pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw"
Nov 03 07:59:27.921168 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:59:27.921138    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw"
Nov 03 07:59:28.191827 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:59:28.191594    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-mdwfw"]
Nov 03 07:59:28.194259 ip-10-0-140-123 kubenswrapper[5070]: W1103 07:59:28.194232    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d26093c_a60a_4cd8_9d43_8ecded62d895.slice/crio-d324fb31646c8413fb50dcba0c4caf6d5f270bad03b3ba7d1f7330d805767c4b WatchSource:0}: Error finding container d324fb31646c8413fb50dcba0c4caf6d5f270bad03b3ba7d1f7330d805767c4b: Status 404 returned error can't find the container with id d324fb31646c8413fb50dcba0c4caf6d5f270bad03b3ba7d1f7330d805767c4b
Nov 03 07:59:28.930473 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:59:28.930440    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw" event={"ID":"0d26093c-a60a-4cd8-9d43-8ecded62d895","Type":"ContainerStarted","Data":"d324fb31646c8413fb50dcba0c4caf6d5f270bad03b3ba7d1f7330d805767c4b"}
Nov 03 07:59:35.976278 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:59:35.976239    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw" event={"ID":"0d26093c-a60a-4cd8-9d43-8ecded62d895","Type":"ContainerStarted","Data":"bfd14b3b1a48099d8a2943d58cb72793f8e8fd272d5702cec8dc340d02fd1f29"}
Nov 03 07:59:35.977274 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:59:35.977251    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw"
Nov 03 07:59:35.991911 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:59:35.991855    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw" podStartSLOduration=1.7662266789999999 podStartE2EDuration="8.991843833s" totalImagesPullingTime="7.225617154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 07:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 07:59:35.989942623 +0000 UTC m=+374.104406565" watchObservedRunningTime="2026-11-03 07:59:35.991843833 +0000 UTC m=+374.106307764"
Nov 03 07:59:46.986488 ip-10-0-140-123 kubenswrapper[5070]: I1103 07:59:46.986455    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw"
Nov 03 07:59:52.051097 ip-10-0-140-123 kubenswrapper[5070]: E1103 07:59:52.051049    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:00:18.643975 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:18.643889    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:00:22.051560 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:00:22.051525    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:00:28.746651 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:28.746615    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ppskm"]
Nov 03 08:00:28.750179 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:28.750160    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-ppskm"
Nov 03 08:00:28.756782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:28.756739    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ppskm"]
Nov 03 08:00:28.760742 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:28.760715    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx8nh\" (UniqueName: \"kubernetes.io/projected/9a690878-a850-481b-b8bc-878c2ab75db0-kube-api-access-dx8nh\") pod \"authorino-6f6944bcdd-ppskm\" (UID: \"9a690878-a850-481b-b8bc-878c2ab75db0\") " pod="kuadrant-system/authorino-6f6944bcdd-ppskm"
Nov 03 08:00:28.848057 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:28.848025    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ppskm"]
Nov 03 08:00:28.848270 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:00:28.848251    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dx8nh], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-ppskm" podUID="9a690878-a850-481b-b8bc-878c2ab75db0"
Nov 03 08:00:28.861629 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:28.861596    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx8nh\" (UniqueName: \"kubernetes.io/projected/9a690878-a850-481b-b8bc-878c2ab75db0-kube-api-access-dx8nh\") pod \"authorino-6f6944bcdd-ppskm\" (UID: \"9a690878-a850-481b-b8bc-878c2ab75db0\") " pod="kuadrant-system/authorino-6f6944bcdd-ppskm"
Nov 03 08:00:28.871786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:28.871747    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-tdcrn"]
Nov 03 08:00:28.874993 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:28.874970    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx8nh\" (UniqueName: \"kubernetes.io/projected/9a690878-a850-481b-b8bc-878c2ab75db0-kube-api-access-dx8nh\") pod \"authorino-6f6944bcdd-ppskm\" (UID: \"9a690878-a850-481b-b8bc-878c2ab75db0\") " pod="kuadrant-system/authorino-6f6944bcdd-ppskm"
Nov 03 08:00:28.875296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:28.875280    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-tdcrn"
Nov 03 08:00:28.882919 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:28.882900    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 03 08:00:28.884155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:28.884128    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-tdcrn"]
Nov 03 08:00:28.962334 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:28.962309    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-976w4\" (UniqueName: \"kubernetes.io/projected/130d4d42-b455-4b96-8ac1-c5b6a80e8e6e-kube-api-access-976w4\") pod \"authorino-6889cbd8c4-tdcrn\" (UID: \"130d4d42-b455-4b96-8ac1-c5b6a80e8e6e\") " pod="kuadrant-system/authorino-6889cbd8c4-tdcrn"
Nov 03 08:00:28.962439 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:28.962381    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/130d4d42-b455-4b96-8ac1-c5b6a80e8e6e-tls-cert\") pod \"authorino-6889cbd8c4-tdcrn\" (UID: \"130d4d42-b455-4b96-8ac1-c5b6a80e8e6e\") " pod="kuadrant-system/authorino-6889cbd8c4-tdcrn"
Nov 03 08:00:29.063296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.063243    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/130d4d42-b455-4b96-8ac1-c5b6a80e8e6e-tls-cert\") pod \"authorino-6889cbd8c4-tdcrn\" (UID: \"130d4d42-b455-4b96-8ac1-c5b6a80e8e6e\") " pod="kuadrant-system/authorino-6889cbd8c4-tdcrn"
Nov 03 08:00:29.063296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.063279    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-976w4\" (UniqueName: \"kubernetes.io/projected/130d4d42-b455-4b96-8ac1-c5b6a80e8e6e-kube-api-access-976w4\") pod \"authorino-6889cbd8c4-tdcrn\" (UID: \"130d4d42-b455-4b96-8ac1-c5b6a80e8e6e\") " pod="kuadrant-system/authorino-6889cbd8c4-tdcrn"
Nov 03 08:00:29.065599 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.065577    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/130d4d42-b455-4b96-8ac1-c5b6a80e8e6e-tls-cert\") pod \"authorino-6889cbd8c4-tdcrn\" (UID: \"130d4d42-b455-4b96-8ac1-c5b6a80e8e6e\") " pod="kuadrant-system/authorino-6889cbd8c4-tdcrn"
Nov 03 08:00:29.070399 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.070378    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-976w4\" (UniqueName: \"kubernetes.io/projected/130d4d42-b455-4b96-8ac1-c5b6a80e8e6e-kube-api-access-976w4\") pod \"authorino-6889cbd8c4-tdcrn\" (UID: \"130d4d42-b455-4b96-8ac1-c5b6a80e8e6e\") " pod="kuadrant-system/authorino-6889cbd8c4-tdcrn"
Nov 03 08:00:29.094507 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.094484    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-tdcrn"]
Nov 03 08:00:29.094640 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.094629    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-tdcrn"
Nov 03 08:00:29.122883 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.122860    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f98ccf99c-lztkr"]
Nov 03 08:00:29.127178 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.127160    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f98ccf99c-lztkr"
Nov 03 08:00:29.136065 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.136039    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f98ccf99c-lztkr"]
Nov 03 08:00:29.164389 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.164357    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bece598b-becd-43a6-a90a-ab8c258bef8d-tls-cert\") pod \"authorino-f98ccf99c-lztkr\" (UID: \"bece598b-becd-43a6-a90a-ab8c258bef8d\") " pod="kuadrant-system/authorino-f98ccf99c-lztkr"
Nov 03 08:00:29.164509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.164388    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfsqk\" (UniqueName: \"kubernetes.io/projected/bece598b-becd-43a6-a90a-ab8c258bef8d-kube-api-access-sfsqk\") pod \"authorino-f98ccf99c-lztkr\" (UID: \"bece598b-becd-43a6-a90a-ab8c258bef8d\") " pod="kuadrant-system/authorino-f98ccf99c-lztkr"
Nov 03 08:00:29.226615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.226592    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-ppskm"
Nov 03 08:00:29.233364 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.233347    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-ppskm"
Nov 03 08:00:29.265125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.265089    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a690878-a850-481b-b8bc-878c2ab75db0-kube-api-access-dx8nh\" (UniqueName: \"kubernetes.io/projected/9a690878-a850-481b-b8bc-878c2ab75db0-kube-api-access-dx8nh\") pod \"9a690878-a850-481b-b8bc-878c2ab75db0\" (UID: \"9a690878-a850-481b-b8bc-878c2ab75db0\") "
Nov 03 08:00:29.265326 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.265305    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bece598b-becd-43a6-a90a-ab8c258bef8d-tls-cert\") pod \"authorino-f98ccf99c-lztkr\" (UID: \"bece598b-becd-43a6-a90a-ab8c258bef8d\") " pod="kuadrant-system/authorino-f98ccf99c-lztkr"
Nov 03 08:00:29.265374 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.265347    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfsqk\" (UniqueName: \"kubernetes.io/projected/bece598b-becd-43a6-a90a-ab8c258bef8d-kube-api-access-sfsqk\") pod \"authorino-f98ccf99c-lztkr\" (UID: \"bece598b-becd-43a6-a90a-ab8c258bef8d\") " pod="kuadrant-system/authorino-f98ccf99c-lztkr"
Nov 03 08:00:29.267387 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.267336    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a690878-a850-481b-b8bc-878c2ab75db0-kube-api-access-dx8nh" pod "9a690878-a850-481b-b8bc-878c2ab75db0" (UID: "9a690878-a850-481b-b8bc-878c2ab75db0"). InnerVolumeSpecName "kube-api-access-dx8nh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:00:29.267903 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.267879    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bece598b-becd-43a6-a90a-ab8c258bef8d-tls-cert\") pod \"authorino-f98ccf99c-lztkr\" (UID: \"bece598b-becd-43a6-a90a-ab8c258bef8d\") " pod="kuadrant-system/authorino-f98ccf99c-lztkr"
Nov 03 08:00:29.272073 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.272056    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfsqk\" (UniqueName: \"kubernetes.io/projected/bece598b-becd-43a6-a90a-ab8c258bef8d-kube-api-access-sfsqk\") pod \"authorino-f98ccf99c-lztkr\" (UID: \"bece598b-becd-43a6-a90a-ab8c258bef8d\") " pod="kuadrant-system/authorino-f98ccf99c-lztkr"
Nov 03 08:00:29.365920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.365892    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-tdcrn"]
Nov 03 08:00:29.366184 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.366163    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx8nh\" (UniqueName: \"kubernetes.io/projected/9a690878-a850-481b-b8bc-878c2ab75db0-kube-api-access-dx8nh\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:00:29.368884 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:00:29.368862    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod130d4d42_b455_4b96_8ac1_c5b6a80e8e6e.slice/crio-6076e636aebf8d363bbc7c957a4e6679a74bfff0ae1b550de9e474e428addaa0 WatchSource:0}: Error finding container 6076e636aebf8d363bbc7c957a4e6679a74bfff0ae1b550de9e474e428addaa0: Status 404 returned error can't find the container with id 6076e636aebf8d363bbc7c957a4e6679a74bfff0ae1b550de9e474e428addaa0
Nov 03 08:00:29.441920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.441900    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f98ccf99c-lztkr"
Nov 03 08:00:29.709753 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:29.709729    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f98ccf99c-lztkr"]
Nov 03 08:00:29.711590 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:00:29.711560    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbece598b_becd_43a6_a90a_ab8c258bef8d.slice/crio-5efab12dcfdcab4909416175ab04d903c0e6eee1d30e80d856788707218191d6 WatchSource:0}: Error finding container 5efab12dcfdcab4909416175ab04d903c0e6eee1d30e80d856788707218191d6: Status 404 returned error can't find the container with id 5efab12dcfdcab4909416175ab04d903c0e6eee1d30e80d856788707218191d6
Nov 03 08:00:30.232574 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:30.232536    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f98ccf99c-lztkr" event={"ID":"bece598b-becd-43a6-a90a-ab8c258bef8d","Type":"ContainerStarted","Data":"5efab12dcfdcab4909416175ab04d903c0e6eee1d30e80d856788707218191d6"}
Nov 03 08:00:30.233863 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:30.233836    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-tdcrn" event={"ID":"130d4d42-b455-4b96-8ac1-c5b6a80e8e6e","Type":"ContainerStarted","Data":"6076e636aebf8d363bbc7c957a4e6679a74bfff0ae1b550de9e474e428addaa0"}
Nov 03 08:00:30.233969 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:30.233867    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-ppskm"
Nov 03 08:00:30.254749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:30.254726    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ppskm"]
Nov 03 08:00:30.258293 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:30.258267    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ppskm"]
Nov 03 08:00:31.239797 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:31.239745    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-tdcrn" event={"ID":"130d4d42-b455-4b96-8ac1-c5b6a80e8e6e","Type":"ContainerStarted","Data":"4b4e3b386746d02329ab11476b176dadfde920696db27fba91b27b624d95e949"}
Nov 03 08:00:31.240244 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:31.239837    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-tdcrn" podUID="130d4d42-b455-4b96-8ac1-c5b6a80e8e6e" containerName="authorino" containerID="cri-o://4b4e3b386746d02329ab11476b176dadfde920696db27fba91b27b624d95e949" gracePeriod=30
Nov 03 08:00:31.241453 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:31.241428    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f98ccf99c-lztkr" event={"ID":"bece598b-becd-43a6-a90a-ab8c258bef8d","Type":"ContainerStarted","Data":"3040f4336f35e3c7dc58fe14ab5e24d41559c502f8b029ef44d701dc90b74ad8"}
Nov 03 08:00:31.255360 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:31.255307    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-tdcrn" podStartSLOduration=2.293528633 podStartE2EDuration="3.25529223s" totalImagesPullingTime="961.763597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:00:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:00:31.254046816 +0000 UTC m=+429.368510747" watchObservedRunningTime="2026-11-03 08:00:31.25529223 +0000 UTC m=+429.369756162"
Nov 03 08:00:31.267251 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:31.267214    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-f98ccf99c-lztkr" podStartSLOduration=1.4577646149999999 podStartE2EDuration="2.267203312s" totalImagesPullingTime="809.438697ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:00:31.266738923 +0000 UTC m=+429.381202874" watchObservedRunningTime="2026-11-03 08:00:31.267203312 +0000 UTC m=+429.381667280"
Nov 03 08:00:31.288953 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:31.288931    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-n2bh7"]
Nov 03 08:00:31.289170 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:31.289114    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-n2bh7" podUID="f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae" containerName="authorino" containerID="cri-o://a110c1cb4b42ec63bdc26ed74f857072e1b3059dbb6d7520b79cdc600e031b47" gracePeriod=30
Nov 03 08:00:31.552190 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:31.552167    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-tdcrn"
Nov 03 08:00:31.555079 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:31.555053    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-n2bh7"
Nov 03 08:00:31.587044 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:31.587019    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/130d4d42-b455-4b96-8ac1-c5b6a80e8e6e-tls-cert\" (UniqueName: \"kubernetes.io/secret/130d4d42-b455-4b96-8ac1-c5b6a80e8e6e-tls-cert\") pod \"130d4d42-b455-4b96-8ac1-c5b6a80e8e6e\" (UID: \"130d4d42-b455-4b96-8ac1-c5b6a80e8e6e\") "
Nov 03 08:00:31.587142 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:31.587057    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/130d4d42-b455-4b96-8ac1-c5b6a80e8e6e-kube-api-access-976w4\" (UniqueName: \"kubernetes.io/projected/130d4d42-b455-4b96-8ac1-c5b6a80e8e6e-kube-api-access-976w4\") pod \"130d4d42-b455-4b96-8ac1-c5b6a80e8e6e\" (UID: \"130d4d42-b455-4b96-8ac1-c5b6a80e8e6e\") "
Nov 03 08:00:31.587142 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:31.587101    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae-kube-api-access-l4sqw\" (UniqueName: \"kubernetes.io/projected/f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae-kube-api-access-l4sqw\") pod \"f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae\" (UID: \"f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae\") "
Nov 03 08:00:31.589537 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:31.589510    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae-kube-api-access-l4sqw" pod "f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae" (UID: "f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae"). InnerVolumeSpecName "kube-api-access-l4sqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:00:31.589867 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:31.589830    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130d4d42-b455-4b96-8ac1-c5b6a80e8e6e-kube-api-access-976w4" pod "130d4d42-b455-4b96-8ac1-c5b6a80e8e6e" (UID: "130d4d42-b455-4b96-8ac1-c5b6a80e8e6e"). InnerVolumeSpecName "kube-api-access-976w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:00:31.598235 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:31.598212    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/130d4d42-b455-4b96-8ac1-c5b6a80e8e6e-tls-cert" pod "130d4d42-b455-4b96-8ac1-c5b6a80e8e6e" (UID: "130d4d42-b455-4b96-8ac1-c5b6a80e8e6e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:00:31.688133 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:31.688109    5070 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/130d4d42-b455-4b96-8ac1-c5b6a80e8e6e-tls-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:00:31.688133 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:31.688133    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-976w4\" (UniqueName: \"kubernetes.io/projected/130d4d42-b455-4b96-8ac1-c5b6a80e8e6e-kube-api-access-976w4\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:00:31.688255 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:31.688143    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4sqw\" (UniqueName: \"kubernetes.io/projected/f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae-kube-api-access-l4sqw\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:00:32.024452 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:32.024421    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a690878-a850-481b-b8bc-878c2ab75db0" path="/var/lib/kubelet/pods/9a690878-a850-481b-b8bc-878c2ab75db0/volumes"
Nov 03 08:00:32.247590 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:32.247513    5070 generic.go:356] "Generic (PLEG): container finished" podID="130d4d42-b455-4b96-8ac1-c5b6a80e8e6e" containerID="4b4e3b386746d02329ab11476b176dadfde920696db27fba91b27b624d95e949" exitCode=0
Nov 03 08:00:32.247590 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:32.247568    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-tdcrn"
Nov 03 08:00:32.248076 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:32.247562    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-tdcrn" event={"ID":"130d4d42-b455-4b96-8ac1-c5b6a80e8e6e","Type":"ContainerDied","Data":"4b4e3b386746d02329ab11476b176dadfde920696db27fba91b27b624d95e949"}
Nov 03 08:00:32.248076 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:32.247690    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-tdcrn" event={"ID":"130d4d42-b455-4b96-8ac1-c5b6a80e8e6e","Type":"ContainerDied","Data":"6076e636aebf8d363bbc7c957a4e6679a74bfff0ae1b550de9e474e428addaa0"}
Nov 03 08:00:32.248076 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:32.247724    5070 scope.go:122] "RemoveContainer" containerID="4b4e3b386746d02329ab11476b176dadfde920696db27fba91b27b624d95e949"
Nov 03 08:00:32.248953 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:32.248931    5070 generic.go:356] "Generic (PLEG): container finished" podID="f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae" containerID="a110c1cb4b42ec63bdc26ed74f857072e1b3059dbb6d7520b79cdc600e031b47" exitCode=0
Nov 03 08:00:32.249059 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:32.248965    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-n2bh7"
Nov 03 08:00:32.249059 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:32.248975    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-n2bh7" event={"ID":"f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae","Type":"ContainerDied","Data":"a110c1cb4b42ec63bdc26ed74f857072e1b3059dbb6d7520b79cdc600e031b47"}
Nov 03 08:00:32.249059 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:32.249026    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-n2bh7" event={"ID":"f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae","Type":"ContainerDied","Data":"10ac802000db080e98a3877b37fcc015f69560eed4c13530247050931f9e165b"}
Nov 03 08:00:32.256516 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:32.256495    5070 scope.go:122] "RemoveContainer" containerID="4b4e3b386746d02329ab11476b176dadfde920696db27fba91b27b624d95e949"
Nov 03 08:00:32.256796 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:00:32.256771    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b4e3b386746d02329ab11476b176dadfde920696db27fba91b27b624d95e949\": container with ID starting with 4b4e3b386746d02329ab11476b176dadfde920696db27fba91b27b624d95e949 not found: ID does not exist" containerID="4b4e3b386746d02329ab11476b176dadfde920696db27fba91b27b624d95e949"
Nov 03 08:00:32.256857 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:32.256807    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b4e3b386746d02329ab11476b176dadfde920696db27fba91b27b624d95e949"} err="failed to get container status \"4b4e3b386746d02329ab11476b176dadfde920696db27fba91b27b624d95e949\": rpc error: code = NotFound desc = could not find container \"4b4e3b386746d02329ab11476b176dadfde920696db27fba91b27b624d95e949\": container with ID starting with 4b4e3b386746d02329ab11476b176dadfde920696db27fba91b27b624d95e949 not found: ID does not exist"
Nov 03 08:00:32.256857 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:32.256825    5070 scope.go:122] "RemoveContainer" containerID="a110c1cb4b42ec63bdc26ed74f857072e1b3059dbb6d7520b79cdc600e031b47"
Nov 03 08:00:32.262775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:32.262742    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-tdcrn"]
Nov 03 08:00:32.265122 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:32.265101    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-tdcrn"]
Nov 03 08:00:32.265964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:32.265936    5070 scope.go:122] "RemoveContainer" containerID="a110c1cb4b42ec63bdc26ed74f857072e1b3059dbb6d7520b79cdc600e031b47"
Nov 03 08:00:32.266233 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:00:32.266215    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a110c1cb4b42ec63bdc26ed74f857072e1b3059dbb6d7520b79cdc600e031b47\": container with ID starting with a110c1cb4b42ec63bdc26ed74f857072e1b3059dbb6d7520b79cdc600e031b47 not found: ID does not exist" containerID="a110c1cb4b42ec63bdc26ed74f857072e1b3059dbb6d7520b79cdc600e031b47"
Nov 03 08:00:32.266289 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:32.266241    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a110c1cb4b42ec63bdc26ed74f857072e1b3059dbb6d7520b79cdc600e031b47"} err="failed to get container status \"a110c1cb4b42ec63bdc26ed74f857072e1b3059dbb6d7520b79cdc600e031b47\": rpc error: code = NotFound desc = could not find container \"a110c1cb4b42ec63bdc26ed74f857072e1b3059dbb6d7520b79cdc600e031b47\": container with ID starting with a110c1cb4b42ec63bdc26ed74f857072e1b3059dbb6d7520b79cdc600e031b47 not found: ID does not exist"
Nov 03 08:00:32.271192 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:32.271172    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-n2bh7"]
Nov 03 08:00:32.274724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:32.274704    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-n2bh7"]
Nov 03 08:00:34.024508 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:34.024472    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="130d4d42-b455-4b96-8ac1-c5b6a80e8e6e" path="/var/lib/kubelet/pods/130d4d42-b455-4b96-8ac1-c5b6a80e8e6e/volumes"
Nov 03 08:00:34.024915 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:00:34.024812    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae" path="/var/lib/kubelet/pods/f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae/volumes"
Nov 03 08:00:52.051089 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:00:52.051048    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:01:22.050525 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:01:22.050488    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:01:22.637032 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:22.636997    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:01:25.429615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.429577    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"]
Nov 03 08:01:25.430125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.430006    5070 cpu_manager.go:414] "containerMap: removing container" podUID="130d4d42-b455-4b96-8ac1-c5b6a80e8e6e" containerName="authorino"
Nov 03 08:01:25.430125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.430020    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="130d4d42-b455-4b96-8ac1-c5b6a80e8e6e" containerName="authorino"
Nov 03 08:01:25.430125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.430033    5070 cpu_manager.go:414] "containerMap: removing container" podUID="f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae" containerName="authorino"
Nov 03 08:01:25.430125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.430037    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae" containerName="authorino"
Nov 03 08:01:25.430125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.430125    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="130d4d42-b455-4b96-8ac1-c5b6a80e8e6e" containerName="authorino"
Nov 03 08:01:25.430366 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.430133    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="f66e6ff0-2de1-44a1-ae9d-52e5b1a40dae" containerName="authorino"
Nov 03 08:01:25.434834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.434815    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:25.440106 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.440081    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 08:01:25.440244 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.440215    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 03 08:01:25.440324 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.440238    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 08:01:25.440324 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.440295    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vcxhj\""
Nov 03 08:01:25.442352 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.442333    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"]
Nov 03 08:01:25.587206 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.587177    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33f324bd-79b1-4540-a090-7e4a9d92f10b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz\" (UID: \"33f324bd-79b1-4540-a090-7e4a9d92f10b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:25.587347 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.587226    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33f324bd-79b1-4540-a090-7e4a9d92f10b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz\" (UID: \"33f324bd-79b1-4540-a090-7e4a9d92f10b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:25.587347 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.587252    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lssnn\" (UniqueName: \"kubernetes.io/projected/33f324bd-79b1-4540-a090-7e4a9d92f10b-kube-api-access-lssnn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz\" (UID: \"33f324bd-79b1-4540-a090-7e4a9d92f10b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:25.587426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.587391    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33f324bd-79b1-4540-a090-7e4a9d92f10b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz\" (UID: \"33f324bd-79b1-4540-a090-7e4a9d92f10b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:25.587464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.587450    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33f324bd-79b1-4540-a090-7e4a9d92f10b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz\" (UID: \"33f324bd-79b1-4540-a090-7e4a9d92f10b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:25.587497 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.587470    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33f324bd-79b1-4540-a090-7e4a9d92f10b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz\" (UID: \"33f324bd-79b1-4540-a090-7e4a9d92f10b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:25.688656 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.688590    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lssnn\" (UniqueName: \"kubernetes.io/projected/33f324bd-79b1-4540-a090-7e4a9d92f10b-kube-api-access-lssnn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz\" (UID: \"33f324bd-79b1-4540-a090-7e4a9d92f10b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:25.688656 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.688649    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33f324bd-79b1-4540-a090-7e4a9d92f10b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz\" (UID: \"33f324bd-79b1-4540-a090-7e4a9d92f10b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:25.688871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.688687    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33f324bd-79b1-4540-a090-7e4a9d92f10b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz\" (UID: \"33f324bd-79b1-4540-a090-7e4a9d92f10b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:25.688871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.688714    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33f324bd-79b1-4540-a090-7e4a9d92f10b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz\" (UID: \"33f324bd-79b1-4540-a090-7e4a9d92f10b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:25.688871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.688778    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33f324bd-79b1-4540-a090-7e4a9d92f10b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz\" (UID: \"33f324bd-79b1-4540-a090-7e4a9d92f10b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:25.688871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.688840    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33f324bd-79b1-4540-a090-7e4a9d92f10b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz\" (UID: \"33f324bd-79b1-4540-a090-7e4a9d92f10b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:25.689114 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.689096    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33f324bd-79b1-4540-a090-7e4a9d92f10b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz\" (UID: \"33f324bd-79b1-4540-a090-7e4a9d92f10b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:25.689183 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.689149    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33f324bd-79b1-4540-a090-7e4a9d92f10b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz\" (UID: \"33f324bd-79b1-4540-a090-7e4a9d92f10b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:25.689239 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.689196    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33f324bd-79b1-4540-a090-7e4a9d92f10b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz\" (UID: \"33f324bd-79b1-4540-a090-7e4a9d92f10b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:25.690931 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.690913    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33f324bd-79b1-4540-a090-7e4a9d92f10b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz\" (UID: \"33f324bd-79b1-4540-a090-7e4a9d92f10b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:25.691209 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.691194    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33f324bd-79b1-4540-a090-7e4a9d92f10b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz\" (UID: \"33f324bd-79b1-4540-a090-7e4a9d92f10b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:25.695145 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.695121    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lssnn\" (UniqueName: \"kubernetes.io/projected/33f324bd-79b1-4540-a090-7e4a9d92f10b-kube-api-access-lssnn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz\" (UID: \"33f324bd-79b1-4540-a090-7e4a9d92f10b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:25.749079 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.749055    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:25.976052 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:25.975979    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:01:26.025660 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.025629    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"]
Nov 03 08:01:26.029169 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:01:26.029140    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f324bd_79b1_4540_a090_7e4a9d92f10b.slice/crio-e14133411b5301280d91de0bcdf08c7f8a0858fe8e49bb5cae42073b79942801 WatchSource:0}: Error finding container e14133411b5301280d91de0bcdf08c7f8a0858fe8e49bb5cae42073b79942801: Status 404 returned error can't find the container with id e14133411b5301280d91de0bcdf08c7f8a0858fe8e49bb5cae42073b79942801
Nov 03 08:01:26.272640 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.272560    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qfg9j"]
Nov 03 08:01:26.277881 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.277851    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:26.286125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.285968    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qfg9j"]
Nov 03 08:01:26.286356 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.286156    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 03 08:01:26.395900 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.395859    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql85h\" (UniqueName: \"kubernetes.io/projected/67eb2350-8895-4e7d-b1f2-e78e1ff5cf23-kube-api-access-ql85h\") pod \"echo-simulator-kserve-64b6f8f96c-qfg9j\" (UID: \"67eb2350-8895-4e7d-b1f2-e78e1ff5cf23\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:26.396062 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.395924    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67eb2350-8895-4e7d-b1f2-e78e1ff5cf23-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qfg9j\" (UID: \"67eb2350-8895-4e7d-b1f2-e78e1ff5cf23\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:26.396062 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.396028    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67eb2350-8895-4e7d-b1f2-e78e1ff5cf23-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qfg9j\" (UID: \"67eb2350-8895-4e7d-b1f2-e78e1ff5cf23\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:26.396181 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.396067    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67eb2350-8895-4e7d-b1f2-e78e1ff5cf23-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qfg9j\" (UID: \"67eb2350-8895-4e7d-b1f2-e78e1ff5cf23\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:26.396181 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.396093    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67eb2350-8895-4e7d-b1f2-e78e1ff5cf23-home\") pod \"echo-simulator-kserve-64b6f8f96c-qfg9j\" (UID: \"67eb2350-8895-4e7d-b1f2-e78e1ff5cf23\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:26.396181 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.396161    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67eb2350-8895-4e7d-b1f2-e78e1ff5cf23-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qfg9j\" (UID: \"67eb2350-8895-4e7d-b1f2-e78e1ff5cf23\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:26.496985 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.496938    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67eb2350-8895-4e7d-b1f2-e78e1ff5cf23-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qfg9j\" (UID: \"67eb2350-8895-4e7d-b1f2-e78e1ff5cf23\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:26.496985 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.496993    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67eb2350-8895-4e7d-b1f2-e78e1ff5cf23-home\") pod \"echo-simulator-kserve-64b6f8f96c-qfg9j\" (UID: \"67eb2350-8895-4e7d-b1f2-e78e1ff5cf23\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:26.497523 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.497010    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67eb2350-8895-4e7d-b1f2-e78e1ff5cf23-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qfg9j\" (UID: \"67eb2350-8895-4e7d-b1f2-e78e1ff5cf23\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:26.497523 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.497062    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql85h\" (UniqueName: \"kubernetes.io/projected/67eb2350-8895-4e7d-b1f2-e78e1ff5cf23-kube-api-access-ql85h\") pod \"echo-simulator-kserve-64b6f8f96c-qfg9j\" (UID: \"67eb2350-8895-4e7d-b1f2-e78e1ff5cf23\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:26.497523 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.497095    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67eb2350-8895-4e7d-b1f2-e78e1ff5cf23-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qfg9j\" (UID: \"67eb2350-8895-4e7d-b1f2-e78e1ff5cf23\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:26.497523 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.497157    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67eb2350-8895-4e7d-b1f2-e78e1ff5cf23-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qfg9j\" (UID: \"67eb2350-8895-4e7d-b1f2-e78e1ff5cf23\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:26.497523 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.497485    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67eb2350-8895-4e7d-b1f2-e78e1ff5cf23-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qfg9j\" (UID: \"67eb2350-8895-4e7d-b1f2-e78e1ff5cf23\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:26.497808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.497659    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67eb2350-8895-4e7d-b1f2-e78e1ff5cf23-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qfg9j\" (UID: \"67eb2350-8895-4e7d-b1f2-e78e1ff5cf23\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:26.497898 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.497881    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67eb2350-8895-4e7d-b1f2-e78e1ff5cf23-home\") pod \"echo-simulator-kserve-64b6f8f96c-qfg9j\" (UID: \"67eb2350-8895-4e7d-b1f2-e78e1ff5cf23\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:26.500520 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.500491    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67eb2350-8895-4e7d-b1f2-e78e1ff5cf23-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qfg9j\" (UID: \"67eb2350-8895-4e7d-b1f2-e78e1ff5cf23\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:26.501527 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.501505    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67eb2350-8895-4e7d-b1f2-e78e1ff5cf23-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qfg9j\" (UID: \"67eb2350-8895-4e7d-b1f2-e78e1ff5cf23\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:26.506007 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.505981    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql85h\" (UniqueName: \"kubernetes.io/projected/67eb2350-8895-4e7d-b1f2-e78e1ff5cf23-kube-api-access-ql85h\") pod \"echo-simulator-kserve-64b6f8f96c-qfg9j\" (UID: \"67eb2350-8895-4e7d-b1f2-e78e1ff5cf23\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:26.545096 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.545008    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz" event={"ID":"33f324bd-79b1-4540-a090-7e4a9d92f10b","Type":"ContainerStarted","Data":"e14133411b5301280d91de0bcdf08c7f8a0858fe8e49bb5cae42073b79942801"}
Nov 03 08:01:26.602137 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.602101    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:26.904717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:26.904351    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qfg9j"]
Nov 03 08:01:27.559032 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:27.558252    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz" event={"ID":"33f324bd-79b1-4540-a090-7e4a9d92f10b","Type":"ContainerStarted","Data":"9fb49bed73b2f616db43d1950dd9bd61f3e9859de81c7e3ee7566695826fb816"}
Nov 03 08:01:27.560508 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:27.560479    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j" event={"ID":"67eb2350-8895-4e7d-b1f2-e78e1ff5cf23","Type":"ContainerStarted","Data":"6c0ca5c0928328ce2ef9a259d94c3a46fc30e870dcbafdba9f291716af7b9f32"}
Nov 03 08:01:27.592436 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:27.592215    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz" podStartSLOduration=1.150965961 podStartE2EDuration="2.592194042s" totalImagesPullingTime="1.441228081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:01:27.588353761 +0000 UTC m=+485.702817689" watchObservedRunningTime="2026-11-03 08:01:27.592194042 +0000 UTC m=+485.706657974"
Nov 03 08:01:27.749374 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:27.749292    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:28.323509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:28.323475    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:01:29.510729 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.510697    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"]
Nov 03 08:01:29.514639 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.514613    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:29.517646 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.517618    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Nov 03 08:01:29.524169 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.524148    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"]
Nov 03 08:01:29.572369 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.572339    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j" event={"ID":"67eb2350-8895-4e7d-b1f2-e78e1ff5cf23","Type":"ContainerStarted","Data":"4b2237a665cca43586d2014014be57a6370769f306fd1c608b91648fda1bbdc7"}
Nov 03 08:01:29.586982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.586941    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j" podStartSLOduration=1.737160803 podStartE2EDuration="3.586925684s" totalImagesPullingTime="1.849764881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:01:29.58653362 +0000 UTC m=+487.700997551" watchObservedRunningTime="2026-11-03 08:01:29.586925684 +0000 UTC m=+487.701389816"
Nov 03 08:01:29.602919 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.602896    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:29.637525 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.637500    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v77mr\" (UniqueName: \"kubernetes.io/projected/66d4513c-da3f-430c-91e3-bb8d782d5917-kube-api-access-v77mr\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t\" (UID: \"66d4513c-da3f-430c-91e3-bb8d782d5917\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:29.637615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.637562    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66d4513c-da3f-430c-91e3-bb8d782d5917-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t\" (UID: \"66d4513c-da3f-430c-91e3-bb8d782d5917\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:29.637685 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.637667    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66d4513c-da3f-430c-91e3-bb8d782d5917-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t\" (UID: \"66d4513c-da3f-430c-91e3-bb8d782d5917\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:29.637726 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.637705    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66d4513c-da3f-430c-91e3-bb8d782d5917-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t\" (UID: \"66d4513c-da3f-430c-91e3-bb8d782d5917\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:29.637795 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.637782    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66d4513c-da3f-430c-91e3-bb8d782d5917-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t\" (UID: \"66d4513c-da3f-430c-91e3-bb8d782d5917\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:29.637836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.637810    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66d4513c-da3f-430c-91e3-bb8d782d5917-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t\" (UID: \"66d4513c-da3f-430c-91e3-bb8d782d5917\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:29.738195 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.738167    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v77mr\" (UniqueName: \"kubernetes.io/projected/66d4513c-da3f-430c-91e3-bb8d782d5917-kube-api-access-v77mr\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t\" (UID: \"66d4513c-da3f-430c-91e3-bb8d782d5917\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:29.738305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.738267    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66d4513c-da3f-430c-91e3-bb8d782d5917-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t\" (UID: \"66d4513c-da3f-430c-91e3-bb8d782d5917\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:29.738305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.738288    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66d4513c-da3f-430c-91e3-bb8d782d5917-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t\" (UID: \"66d4513c-da3f-430c-91e3-bb8d782d5917\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:29.738395 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.738329    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66d4513c-da3f-430c-91e3-bb8d782d5917-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t\" (UID: \"66d4513c-da3f-430c-91e3-bb8d782d5917\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:29.738395 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.738388    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66d4513c-da3f-430c-91e3-bb8d782d5917-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t\" (UID: \"66d4513c-da3f-430c-91e3-bb8d782d5917\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:29.738496 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.738438    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66d4513c-da3f-430c-91e3-bb8d782d5917-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t\" (UID: \"66d4513c-da3f-430c-91e3-bb8d782d5917\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:29.738747 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.738717    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66d4513c-da3f-430c-91e3-bb8d782d5917-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t\" (UID: \"66d4513c-da3f-430c-91e3-bb8d782d5917\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:29.738887 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.738805    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66d4513c-da3f-430c-91e3-bb8d782d5917-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t\" (UID: \"66d4513c-da3f-430c-91e3-bb8d782d5917\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:29.738887 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.738847    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66d4513c-da3f-430c-91e3-bb8d782d5917-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t\" (UID: \"66d4513c-da3f-430c-91e3-bb8d782d5917\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:29.740739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.740718    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66d4513c-da3f-430c-91e3-bb8d782d5917-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t\" (UID: \"66d4513c-da3f-430c-91e3-bb8d782d5917\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:29.741039 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.741022    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66d4513c-da3f-430c-91e3-bb8d782d5917-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t\" (UID: \"66d4513c-da3f-430c-91e3-bb8d782d5917\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:29.745169 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.745145    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v77mr\" (UniqueName: \"kubernetes.io/projected/66d4513c-da3f-430c-91e3-bb8d782d5917-kube-api-access-v77mr\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t\" (UID: \"66d4513c-da3f-430c-91e3-bb8d782d5917\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:29.829389 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:29.829320    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:30.099696 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.099617    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"]
Nov 03 08:01:30.102684 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:01:30.102655    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66d4513c_da3f_430c_91e3_bb8d782d5917.slice/crio-4d5fc97ebf15cf6c214107d7757fe0775feacde629f960b98a80ffca0f90ab20 WatchSource:0}: Error finding container 4d5fc97ebf15cf6c214107d7757fe0775feacde629f960b98a80ffca0f90ab20: Status 404 returned error can't find the container with id 4d5fc97ebf15cf6c214107d7757fe0775feacde629f960b98a80ffca0f90ab20
Nov 03 08:01:30.263458 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.263428    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:01:30.359007 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.358979    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"]
Nov 03 08:01:30.363949 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.363926    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:30.370906 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.370882    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 03 08:01:30.373603 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.373570    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"]
Nov 03 08:01:30.443934 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.443894    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d50d86f3-08a1-419c-a77b-d7223e0567c7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4\" (UID: \"d50d86f3-08a1-419c-a77b-d7223e0567c7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:30.444109 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.443939    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d50d86f3-08a1-419c-a77b-d7223e0567c7-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4\" (UID: \"d50d86f3-08a1-419c-a77b-d7223e0567c7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:30.444220 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.444192    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d50d86f3-08a1-419c-a77b-d7223e0567c7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4\" (UID: \"d50d86f3-08a1-419c-a77b-d7223e0567c7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:30.444280 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.444257    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqtsw\" (UniqueName: \"kubernetes.io/projected/d50d86f3-08a1-419c-a77b-d7223e0567c7-kube-api-access-cqtsw\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4\" (UID: \"d50d86f3-08a1-419c-a77b-d7223e0567c7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:30.444365 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.444344    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d50d86f3-08a1-419c-a77b-d7223e0567c7-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4\" (UID: \"d50d86f3-08a1-419c-a77b-d7223e0567c7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:30.444463 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.444441    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50d86f3-08a1-419c-a77b-d7223e0567c7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4\" (UID: \"d50d86f3-08a1-419c-a77b-d7223e0567c7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:30.545375 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.545344    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50d86f3-08a1-419c-a77b-d7223e0567c7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4\" (UID: \"d50d86f3-08a1-419c-a77b-d7223e0567c7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:30.545904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.545419    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d50d86f3-08a1-419c-a77b-d7223e0567c7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4\" (UID: \"d50d86f3-08a1-419c-a77b-d7223e0567c7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:30.545904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.545439    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d50d86f3-08a1-419c-a77b-d7223e0567c7-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4\" (UID: \"d50d86f3-08a1-419c-a77b-d7223e0567c7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:30.545904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.545466    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d50d86f3-08a1-419c-a77b-d7223e0567c7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4\" (UID: \"d50d86f3-08a1-419c-a77b-d7223e0567c7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:30.545904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.545512    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqtsw\" (UniqueName: \"kubernetes.io/projected/d50d86f3-08a1-419c-a77b-d7223e0567c7-kube-api-access-cqtsw\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4\" (UID: \"d50d86f3-08a1-419c-a77b-d7223e0567c7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:30.545904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.545543    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d50d86f3-08a1-419c-a77b-d7223e0567c7-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4\" (UID: \"d50d86f3-08a1-419c-a77b-d7223e0567c7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:30.545904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.545852    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d50d86f3-08a1-419c-a77b-d7223e0567c7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4\" (UID: \"d50d86f3-08a1-419c-a77b-d7223e0567c7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:30.546244 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.545930    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d50d86f3-08a1-419c-a77b-d7223e0567c7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4\" (UID: \"d50d86f3-08a1-419c-a77b-d7223e0567c7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:30.546244 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.546192    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d50d86f3-08a1-419c-a77b-d7223e0567c7-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4\" (UID: \"d50d86f3-08a1-419c-a77b-d7223e0567c7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:30.547803 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.547748    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d50d86f3-08a1-419c-a77b-d7223e0567c7-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4\" (UID: \"d50d86f3-08a1-419c-a77b-d7223e0567c7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:30.548057 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.548018    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50d86f3-08a1-419c-a77b-d7223e0567c7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4\" (UID: \"d50d86f3-08a1-419c-a77b-d7223e0567c7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:30.558224 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.558198    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqtsw\" (UniqueName: \"kubernetes.io/projected/d50d86f3-08a1-419c-a77b-d7223e0567c7-kube-api-access-cqtsw\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4\" (UID: \"d50d86f3-08a1-419c-a77b-d7223e0567c7\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:30.580406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.579610    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" event={"ID":"66d4513c-da3f-430c-91e3-bb8d782d5917","Type":"ContainerStarted","Data":"23894544b1b242e2fe3844f9005fc6034d608ccf5f64773175f17a8259d02272"}
Nov 03 08:01:30.580406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.579676    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" event={"ID":"66d4513c-da3f-430c-91e3-bb8d782d5917","Type":"ContainerStarted","Data":"4d5fc97ebf15cf6c214107d7757fe0775feacde629f960b98a80ffca0f90ab20"}
Nov 03 08:01:30.595256 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.595202    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podStartSLOduration=1.227016241 podStartE2EDuration="1.595188235s" totalImagesPullingTime="368.171994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:01:30.594554117 +0000 UTC m=+488.709018049" watchObservedRunningTime="2026-11-03 08:01:30.595188235 +0000 UTC m=+488.709652160"
Nov 03 08:01:30.688245 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.687924    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:30.830251 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.830217    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:30.985475 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:30.985451    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"]
Nov 03 08:01:30.988222 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:01:30.988194    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50d86f3_08a1_419c_a77b_d7223e0567c7.slice/crio-cc1960e9c6ce8c8fad8f9c8cb5c696dd744e8350eddd565fb4b06aee2b069e91 WatchSource:0}: Error finding container cc1960e9c6ce8c8fad8f9c8cb5c696dd744e8350eddd565fb4b06aee2b069e91: Status 404 returned error can't find the container with id cc1960e9c6ce8c8fad8f9c8cb5c696dd744e8350eddd565fb4b06aee2b069e91
Nov 03 08:01:31.594347 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:31.594263    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4" event={"ID":"d50d86f3-08a1-419c-a77b-d7223e0567c7","Type":"ContainerStarted","Data":"aa41a74b89ca370acda70b8491bc51d22a7159af7cec0a19e32be7376241fd27"}
Nov 03 08:01:31.594749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:31.594357    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4" event={"ID":"d50d86f3-08a1-419c-a77b-d7223e0567c7","Type":"ContainerStarted","Data":"cc1960e9c6ce8c8fad8f9c8cb5c696dd744e8350eddd565fb4b06aee2b069e91"}
Nov 03 08:01:31.614780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:31.614721    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4" podStartSLOduration=1.280306974 podStartE2EDuration="1.614703242s" totalImagesPullingTime="334.396268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:01:31.61357407 +0000 UTC m=+489.728038001" watchObservedRunningTime="2026-11-03 08:01:31.614703242 +0000 UTC m=+489.729167175"
Nov 03 08:01:31.689378 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:31.689351    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:33.214477 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.214382    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"]
Nov 03 08:01:33.221298 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.221273    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:33.226146 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.226117    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"]
Nov 03 08:01:33.226327 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.226121    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 03 08:01:33.370311 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.370278    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6960b432-0c5c-4c03-bcc5-dec9ff9466c7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gstjn\" (UID: \"6960b432-0c5c-4c03-bcc5-dec9ff9466c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:33.370311 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.370315    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6960b432-0c5c-4c03-bcc5-dec9ff9466c7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gstjn\" (UID: \"6960b432-0c5c-4c03-bcc5-dec9ff9466c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:33.370533 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.370340    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6960b432-0c5c-4c03-bcc5-dec9ff9466c7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gstjn\" (UID: \"6960b432-0c5c-4c03-bcc5-dec9ff9466c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:33.370582 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.370524    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6960b432-0c5c-4c03-bcc5-dec9ff9466c7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gstjn\" (UID: \"6960b432-0c5c-4c03-bcc5-dec9ff9466c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:33.370634 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.370616    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6960b432-0c5c-4c03-bcc5-dec9ff9466c7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gstjn\" (UID: \"6960b432-0c5c-4c03-bcc5-dec9ff9466c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:33.370688 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.370669    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb5c9\" (UniqueName: \"kubernetes.io/projected/6960b432-0c5c-4c03-bcc5-dec9ff9466c7-kube-api-access-jb5c9\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gstjn\" (UID: \"6960b432-0c5c-4c03-bcc5-dec9ff9466c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:33.472103 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.472025    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6960b432-0c5c-4c03-bcc5-dec9ff9466c7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gstjn\" (UID: \"6960b432-0c5c-4c03-bcc5-dec9ff9466c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:33.472103 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.472076    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb5c9\" (UniqueName: \"kubernetes.io/projected/6960b432-0c5c-4c03-bcc5-dec9ff9466c7-kube-api-access-jb5c9\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gstjn\" (UID: \"6960b432-0c5c-4c03-bcc5-dec9ff9466c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:33.472304 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.472124    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6960b432-0c5c-4c03-bcc5-dec9ff9466c7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gstjn\" (UID: \"6960b432-0c5c-4c03-bcc5-dec9ff9466c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:33.472304 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.472140    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6960b432-0c5c-4c03-bcc5-dec9ff9466c7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gstjn\" (UID: \"6960b432-0c5c-4c03-bcc5-dec9ff9466c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:33.472304 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.472160    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6960b432-0c5c-4c03-bcc5-dec9ff9466c7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gstjn\" (UID: \"6960b432-0c5c-4c03-bcc5-dec9ff9466c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:33.472304 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.472185    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6960b432-0c5c-4c03-bcc5-dec9ff9466c7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gstjn\" (UID: \"6960b432-0c5c-4c03-bcc5-dec9ff9466c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:33.472668 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.472640    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6960b432-0c5c-4c03-bcc5-dec9ff9466c7-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gstjn\" (UID: \"6960b432-0c5c-4c03-bcc5-dec9ff9466c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:33.472814 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.472643    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6960b432-0c5c-4c03-bcc5-dec9ff9466c7-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gstjn\" (UID: \"6960b432-0c5c-4c03-bcc5-dec9ff9466c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:33.472814 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.472771    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6960b432-0c5c-4c03-bcc5-dec9ff9466c7-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gstjn\" (UID: \"6960b432-0c5c-4c03-bcc5-dec9ff9466c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:33.474585 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.474558    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6960b432-0c5c-4c03-bcc5-dec9ff9466c7-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gstjn\" (UID: \"6960b432-0c5c-4c03-bcc5-dec9ff9466c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:33.474850 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.474830    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6960b432-0c5c-4c03-bcc5-dec9ff9466c7-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gstjn\" (UID: \"6960b432-0c5c-4c03-bcc5-dec9ff9466c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:33.479459 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.479438    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb5c9\" (UniqueName: \"kubernetes.io/projected/6960b432-0c5c-4c03-bcc5-dec9ff9466c7-kube-api-access-jb5c9\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-gstjn\" (UID: \"6960b432-0c5c-4c03-bcc5-dec9ff9466c7\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:33.539020 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.538995    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:33.815124 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:33.815097    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"]
Nov 03 08:01:33.816412 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:01:33.816371    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6960b432_0c5c_4c03_bcc5_dec9ff9466c7.slice/crio-7b4e72dc741627faaa21c7b94241206c98d17500175a26d5756045d98b43de4a WatchSource:0}: Error finding container 7b4e72dc741627faaa21c7b94241206c98d17500175a26d5756045d98b43de4a: Status 404 returned error can't find the container with id 7b4e72dc741627faaa21c7b94241206c98d17500175a26d5756045d98b43de4a
Nov 03 08:01:34.618351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:34.618323    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn" event={"ID":"6960b432-0c5c-4c03-bcc5-dec9ff9466c7","Type":"ContainerStarted","Data":"d97a28880d3d4357b89727285cff9af4745cce4d231b67b476f01d60d27e43b5"}
Nov 03 08:01:34.618696 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:34.618374    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn" event={"ID":"6960b432-0c5c-4c03-bcc5-dec9ff9466c7","Type":"ContainerStarted","Data":"7b4e72dc741627faaa21c7b94241206c98d17500175a26d5756045d98b43de4a"}
Nov 03 08:01:34.634806 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:34.634747    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn" podStartSLOduration=1.314258578 podStartE2EDuration="1.634735204s" totalImagesPullingTime="320.476626ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:01:34.632396451 +0000 UTC m=+492.746860381" watchObservedRunningTime="2026-11-03 08:01:34.634735204 +0000 UTC m=+492.749199134"
Nov 03 08:01:35.469356 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:35.469328    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:01:35.540149 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:35.540116    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:35.750128 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:35.750052    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:35.768273 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:35.768246    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:36.602382 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:36.602340    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:36.614698 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:36.614675    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:36.641173 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:36.641150    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:01:36.641987 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:36.641971    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:01:39.829737 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:39.829706    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:39.842150 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:39.842126    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:40.662031 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:40.662003    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:01:40.689569 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:40.689537    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:40.701948 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:40.701922    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:41.666676 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:41.666645    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:01:43.539979 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:43.539944    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:43.552088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:43.552066    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:43.677074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:43.677052    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:01:47.702460 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:47.702424    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6995574766-s7dx9"]
Nov 03 08:01:47.706152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:47.706133    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6995574766-s7dx9"
Nov 03 08:01:47.712558 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:47.712532    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6995574766-s7dx9"]
Nov 03 08:01:47.795913 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:47.795884    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b148fccf-b3ae-4b52-9df1-21e85f7e3fa3-tls-cert\") pod \"authorino-6995574766-s7dx9\" (UID: \"b148fccf-b3ae-4b52-9df1-21e85f7e3fa3\") " pod="kuadrant-system/authorino-6995574766-s7dx9"
Nov 03 08:01:47.796041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:47.795922    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f5dg\" (UniqueName: \"kubernetes.io/projected/b148fccf-b3ae-4b52-9df1-21e85f7e3fa3-kube-api-access-9f5dg\") pod \"authorino-6995574766-s7dx9\" (UID: \"b148fccf-b3ae-4b52-9df1-21e85f7e3fa3\") " pod="kuadrant-system/authorino-6995574766-s7dx9"
Nov 03 08:01:47.896296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:47.896263    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b148fccf-b3ae-4b52-9df1-21e85f7e3fa3-tls-cert\") pod \"authorino-6995574766-s7dx9\" (UID: \"b148fccf-b3ae-4b52-9df1-21e85f7e3fa3\") " pod="kuadrant-system/authorino-6995574766-s7dx9"
Nov 03 08:01:47.896421 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:47.896310    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f5dg\" (UniqueName: \"kubernetes.io/projected/b148fccf-b3ae-4b52-9df1-21e85f7e3fa3-kube-api-access-9f5dg\") pod \"authorino-6995574766-s7dx9\" (UID: \"b148fccf-b3ae-4b52-9df1-21e85f7e3fa3\") " pod="kuadrant-system/authorino-6995574766-s7dx9"
Nov 03 08:01:47.898704 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:47.898677    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b148fccf-b3ae-4b52-9df1-21e85f7e3fa3-tls-cert\") pod \"authorino-6995574766-s7dx9\" (UID: \"b148fccf-b3ae-4b52-9df1-21e85f7e3fa3\") " pod="kuadrant-system/authorino-6995574766-s7dx9"
Nov 03 08:01:47.902940 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:47.902919    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f5dg\" (UniqueName: \"kubernetes.io/projected/b148fccf-b3ae-4b52-9df1-21e85f7e3fa3-kube-api-access-9f5dg\") pod \"authorino-6995574766-s7dx9\" (UID: \"b148fccf-b3ae-4b52-9df1-21e85f7e3fa3\") " pod="kuadrant-system/authorino-6995574766-s7dx9"
Nov 03 08:01:48.020438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:48.020388    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6995574766-s7dx9"
Nov 03 08:01:48.288398 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:48.288366    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6995574766-s7dx9"]
Nov 03 08:01:48.290163 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:01:48.290136    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb148fccf_b3ae_4b52_9df1_21e85f7e3fa3.slice/crio-cc922082b8337e28f8074392ee472990ead72953bdda77b9d9bbde1b520dc956 WatchSource:0}: Error finding container cc922082b8337e28f8074392ee472990ead72953bdda77b9d9bbde1b520dc956: Status 404 returned error can't find the container with id cc922082b8337e28f8074392ee472990ead72953bdda77b9d9bbde1b520dc956
Nov 03 08:01:48.694916 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:48.694887    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6995574766-s7dx9" event={"ID":"b148fccf-b3ae-4b52-9df1-21e85f7e3fa3","Type":"ContainerStarted","Data":"cc922082b8337e28f8074392ee472990ead72953bdda77b9d9bbde1b520dc956"}
Nov 03 08:01:49.708405 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:49.708360    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6995574766-s7dx9" event={"ID":"b148fccf-b3ae-4b52-9df1-21e85f7e3fa3","Type":"ContainerStarted","Data":"463db48a813e80996804219635ee41e5d819d37aff48f4197230cb43f6a86ad7"}
Nov 03 08:01:49.724122 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:49.724062    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6995574766-s7dx9" podStartSLOduration=1.895595406 podStartE2EDuration="2.72404775s" totalImagesPullingTime="828.452344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:01:49.720669922 +0000 UTC m=+507.835133857" watchObservedRunningTime="2026-11-03 08:01:49.72404775 +0000 UTC m=+507.838511680"
Nov 03 08:01:49.746918 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:49.746882    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f98ccf99c-lztkr"]
Nov 03 08:01:49.747103 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:49.747075    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-f98ccf99c-lztkr" podUID="bece598b-becd-43a6-a90a-ab8c258bef8d" containerName="authorino" containerID="cri-o://3040f4336f35e3c7dc58fe14ab5e24d41559c502f8b029ef44d701dc90b74ad8" gracePeriod=30
Nov 03 08:01:50.001599 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:50.001569    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f98ccf99c-lztkr"
Nov 03 08:01:50.012793 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:50.011855    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bece598b-becd-43a6-a90a-ab8c258bef8d-tls-cert\" (UniqueName: \"kubernetes.io/secret/bece598b-becd-43a6-a90a-ab8c258bef8d-tls-cert\") pod \"bece598b-becd-43a6-a90a-ab8c258bef8d\" (UID: \"bece598b-becd-43a6-a90a-ab8c258bef8d\") "
Nov 03 08:01:50.012793 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:50.011957    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bece598b-becd-43a6-a90a-ab8c258bef8d-kube-api-access-sfsqk\" (UniqueName: \"kubernetes.io/projected/bece598b-becd-43a6-a90a-ab8c258bef8d-kube-api-access-sfsqk\") pod \"bece598b-becd-43a6-a90a-ab8c258bef8d\" (UID: \"bece598b-becd-43a6-a90a-ab8c258bef8d\") "
Nov 03 08:01:50.020657 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:50.018313    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bece598b-becd-43a6-a90a-ab8c258bef8d-kube-api-access-sfsqk" pod "bece598b-becd-43a6-a90a-ab8c258bef8d" (UID: "bece598b-becd-43a6-a90a-ab8c258bef8d"). InnerVolumeSpecName "kube-api-access-sfsqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:01:50.030908 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:50.030876    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bece598b-becd-43a6-a90a-ab8c258bef8d-tls-cert" pod "bece598b-becd-43a6-a90a-ab8c258bef8d" (UID: "bece598b-becd-43a6-a90a-ab8c258bef8d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:01:50.113130 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:50.113090    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfsqk\" (UniqueName: \"kubernetes.io/projected/bece598b-becd-43a6-a90a-ab8c258bef8d-kube-api-access-sfsqk\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:01:50.113130 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:50.113117    5070 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bece598b-becd-43a6-a90a-ab8c258bef8d-tls-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:01:50.715121 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:50.715085    5070 generic.go:356] "Generic (PLEG): container finished" podID="bece598b-becd-43a6-a90a-ab8c258bef8d" containerID="3040f4336f35e3c7dc58fe14ab5e24d41559c502f8b029ef44d701dc90b74ad8" exitCode=0
Nov 03 08:01:50.715562 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:50.715133    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f98ccf99c-lztkr"
Nov 03 08:01:50.715562 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:50.715171    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f98ccf99c-lztkr" event={"ID":"bece598b-becd-43a6-a90a-ab8c258bef8d","Type":"ContainerDied","Data":"3040f4336f35e3c7dc58fe14ab5e24d41559c502f8b029ef44d701dc90b74ad8"}
Nov 03 08:01:50.715562 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:50.715225    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f98ccf99c-lztkr" event={"ID":"bece598b-becd-43a6-a90a-ab8c258bef8d","Type":"ContainerDied","Data":"5efab12dcfdcab4909416175ab04d903c0e6eee1d30e80d856788707218191d6"}
Nov 03 08:01:50.715562 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:50.715251    5070 scope.go:122] "RemoveContainer" containerID="3040f4336f35e3c7dc58fe14ab5e24d41559c502f8b029ef44d701dc90b74ad8"
Nov 03 08:01:50.724585 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:50.724559    5070 scope.go:122] "RemoveContainer" containerID="3040f4336f35e3c7dc58fe14ab5e24d41559c502f8b029ef44d701dc90b74ad8"
Nov 03 08:01:50.724920 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:01:50.724901    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3040f4336f35e3c7dc58fe14ab5e24d41559c502f8b029ef44d701dc90b74ad8\": container with ID starting with 3040f4336f35e3c7dc58fe14ab5e24d41559c502f8b029ef44d701dc90b74ad8 not found: ID does not exist" containerID="3040f4336f35e3c7dc58fe14ab5e24d41559c502f8b029ef44d701dc90b74ad8"
Nov 03 08:01:50.724997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:50.724932    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3040f4336f35e3c7dc58fe14ab5e24d41559c502f8b029ef44d701dc90b74ad8"} err="failed to get container status \"3040f4336f35e3c7dc58fe14ab5e24d41559c502f8b029ef44d701dc90b74ad8\": rpc error: code = NotFound desc = could not find container \"3040f4336f35e3c7dc58fe14ab5e24d41559c502f8b029ef44d701dc90b74ad8\": container with ID starting with 3040f4336f35e3c7dc58fe14ab5e24d41559c502f8b029ef44d701dc90b74ad8 not found: ID does not exist"
Nov 03 08:01:50.734513 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:50.734488    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f98ccf99c-lztkr"]
Nov 03 08:01:50.738125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:50.738104    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-f98ccf99c-lztkr"]
Nov 03 08:01:51.168970 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.168936    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"]
Nov 03 08:01:51.169383 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.169367    5070 cpu_manager.go:414] "containerMap: removing container" podUID="bece598b-becd-43a6-a90a-ab8c258bef8d" containerName="authorino"
Nov 03 08:01:51.169383 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.169383    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bece598b-becd-43a6-a90a-ab8c258bef8d" containerName="authorino"
Nov 03 08:01:51.169522 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.169477    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="bece598b-becd-43a6-a90a-ab8c258bef8d" containerName="authorino"
Nov 03 08:01:51.172490 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.172468    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.176598 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.176400    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 08:01:51.178061 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.178039    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zg2vp\""
Nov 03 08:01:51.191188 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.191162    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"]
Nov 03 08:01:51.222955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.222909    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c16356-5579-4aac-aec8-c72808dbb1a2-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.222955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.222944    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c16356-5579-4aac-aec8-c72808dbb1a2-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.223182 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.222969    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4c16356-5579-4aac-aec8-c72808dbb1a2-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.223182 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.223088    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4c16356-5579-4aac-aec8-c72808dbb1a2-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.223182 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.223126    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4c16356-5579-4aac-aec8-c72808dbb1a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.223182 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.223143    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4c16356-5579-4aac-aec8-c72808dbb1a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.223402 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.223188    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4c16356-5579-4aac-aec8-c72808dbb1a2-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.223402 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.223324    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4c16356-5579-4aac-aec8-c72808dbb1a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.223402 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.223358    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb82f\" (UniqueName: \"kubernetes.io/projected/c4c16356-5579-4aac-aec8-c72808dbb1a2-kube-api-access-kb82f\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.323920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.323873    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4c16356-5579-4aac-aec8-c72808dbb1a2-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.323920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.323923    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4c16356-5579-4aac-aec8-c72808dbb1a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.324166 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.324035    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4c16356-5579-4aac-aec8-c72808dbb1a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.324166 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.324076    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4c16356-5579-4aac-aec8-c72808dbb1a2-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.324166 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.324148    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4c16356-5579-4aac-aec8-c72808dbb1a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.324325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.324176    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb82f\" (UniqueName: \"kubernetes.io/projected/c4c16356-5579-4aac-aec8-c72808dbb1a2-kube-api-access-kb82f\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.324325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.324228    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c16356-5579-4aac-aec8-c72808dbb1a2-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.324325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.324259    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c16356-5579-4aac-aec8-c72808dbb1a2-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.324325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.324277    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4c16356-5579-4aac-aec8-c72808dbb1a2-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.324325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.324302    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4c16356-5579-4aac-aec8-c72808dbb1a2-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.324577 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.324550    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c16356-5579-4aac-aec8-c72808dbb1a2-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.324773 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.324733    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4c16356-5579-4aac-aec8-c72808dbb1a2-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.324851 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.324801    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4c16356-5579-4aac-aec8-c72808dbb1a2-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.325154 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.325123    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4c16356-5579-4aac-aec8-c72808dbb1a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.326293 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.326270    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4c16356-5579-4aac-aec8-c72808dbb1a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.326649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.326628    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4c16356-5579-4aac-aec8-c72808dbb1a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.331054 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.331032    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4c16356-5579-4aac-aec8-c72808dbb1a2-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.331486 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.331459    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb82f\" (UniqueName: \"kubernetes.io/projected/c4c16356-5579-4aac-aec8-c72808dbb1a2-kube-api-access-kb82f\") pod \"maas-default-gateway-openshift-default-76f4876667-ln9hl\" (UID: \"c4c16356-5579-4aac-aec8-c72808dbb1a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.498688 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.498608    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:51.783115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:51.783085    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"]
Nov 03 08:01:51.785320 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:01:51.785284    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c16356_5579_4aac_aec8_c72808dbb1a2.slice/crio-0ea6990e8e553e38d455f7c186f064b217df4b8bc8ff1d692305897893dbee96 WatchSource:0}: Error finding container 0ea6990e8e553e38d455f7c186f064b217df4b8bc8ff1d692305897893dbee96: Status 404 returned error can't find the container with id 0ea6990e8e553e38d455f7c186f064b217df4b8bc8ff1d692305897893dbee96
Nov 03 08:01:52.025303 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:52.025267    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bece598b-becd-43a6-a90a-ab8c258bef8d" path="/var/lib/kubelet/pods/bece598b-becd-43a6-a90a-ab8c258bef8d/volumes"
Nov 03 08:01:52.050793 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:01:52.050699    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:01:52.728238 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:52.728206    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl" event={"ID":"c4c16356-5579-4aac-aec8-c72808dbb1a2","Type":"ContainerStarted","Data":"0ea6990e8e553e38d455f7c186f064b217df4b8bc8ff1d692305897893dbee96"}
Nov 03 08:01:56.007082 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:56.007048    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:01:56.007368 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:56.007120    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:01:56.007368 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:56.007148    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:01:56.753045 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:56.753011    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl" event={"ID":"c4c16356-5579-4aac-aec8-c72808dbb1a2","Type":"ContainerStarted","Data":"5d6b4f7ac77ba5646aa0270c8905e30f747529e1144329cc8a6464cd1d9b21ab"}
Nov 03 08:01:56.796553 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:56.796502    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl" podStartSLOduration=1.578410285 podStartE2EDuration="5.796489677s" totalImagesPullingTime="4.218079392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:01:56.794485387 +0000 UTC m=+514.908949319" watchObservedRunningTime="2026-11-03 08:01:56.796489677 +0000 UTC m=+514.910953608"
Nov 03 08:01:57.499359 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:57.499321    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:01:57.500720 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:57.500694    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-ln9hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 08:01:57.500834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:57.500751    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl" podUID="c4c16356-5579-4aac-aec8-c72808dbb1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 08:01:58.499520 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:58.499489    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-ln9hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 08:01:58.499935 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:58.499541    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl" podUID="c4c16356-5579-4aac-aec8-c72808dbb1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 08:01:59.499729 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:59.499695    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-ln9hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 08:01:59.500222 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:01:59.499750    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl" podUID="c4c16356-5579-4aac-aec8-c72808dbb1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 08:02:00.499232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:00.499196    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-ln9hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 08:02:00.499410 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:00.499251    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl" podUID="c4c16356-5579-4aac-aec8-c72808dbb1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 08:02:01.499844 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:01.499808    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-ln9hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 08:02:01.500320 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:01.499868    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl" podUID="c4c16356-5579-4aac-aec8-c72808dbb1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 08:02:02.499808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:02.499743    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-ln9hl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Nov 03 08:02:02.500009 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:02.499823    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl" podUID="c4c16356-5579-4aac-aec8-c72808dbb1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Nov 03 08:02:04.209746 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:04.209716    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:02:04.210463 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:04.210440    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:02:04.211113 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:04.211095    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl"
Nov 03 08:02:06.190669 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.190628    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"]
Nov 03 08:02:06.197458 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.197429    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.205509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.205128    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"]
Nov 03 08:02:06.221532 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.221506    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"]
Nov 03 08:02:06.225378 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.225360    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.236665 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.236645    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"]
Nov 03 08:02:06.265870 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.265851    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rspl\" (UniqueName: \"kubernetes.io/projected/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-kube-api-access-4rspl\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.265968 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.265876    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.265968 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.265894    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.265968 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.265911    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.266082 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.265984    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.266082 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.266032    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.266082 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.266060    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.266186 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.266156    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.266222 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.266183    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.266222 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.266203    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.266294 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.266241    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.266294 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.266266    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337a39e3-b285-4d8c-aa1e-210a6e46763a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.266364 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.266337    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.266403 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.266363    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.266438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.266426    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.266483 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.266459    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.266539 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.266526    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.266576 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.266567    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmj52\" (UniqueName: \"kubernetes.io/projected/337a39e3-b285-4d8c-aa1e-210a6e46763a-kube-api-access-jmj52\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.367977 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.367954    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.368079 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.367981    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.368079 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.367998    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.368079 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.368020    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.368229 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.368074    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.368286 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.368257    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.368346 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.368297    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.368346 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.368332    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.368456 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.368366    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.368456 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.368408    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.368569 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.368457    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337a39e3-b285-4d8c-aa1e-210a6e46763a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.368569 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.368511    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.368569 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.368552    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.368717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.368597    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.368717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.368623    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.368717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.368631    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.368717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.368642    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.368717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.368701    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.369049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.368726    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.369049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.368749    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmj52\" (UniqueName: \"kubernetes.io/projected/337a39e3-b285-4d8c-aa1e-210a6e46763a-kube-api-access-jmj52\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.369049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.368847    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rspl\" (UniqueName: \"kubernetes.io/projected/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-kube-api-access-4rspl\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.369049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.368879    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.369049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.368885    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.369325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.369248    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.369427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.369401    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.369609 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.369586    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.369680 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.369643    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337a39e3-b285-4d8c-aa1e-210a6e46763a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.370179 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.370155    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.370744 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.370723    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.370881 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.370857    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.370947 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.370878    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.371046 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.371027    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.375004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.374979    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmj52\" (UniqueName: \"kubernetes.io/projected/337a39e3-b285-4d8c-aa1e-210a6e46763a-kube-api-access-jmj52\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.375111 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.375039    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-tlbnz\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.375111 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.375077    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.375399 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.375380    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rspl\" (UniqueName: \"kubernetes.io/projected/0c9ad9ff-3d0f-4994-829c-81fe102cc3de-kube-api-access-4rspl\") pod \"maas-default-gateway-openshift-default-76f4876667-87snj\" (UID: \"0c9ad9ff-3d0f-4994-829c-81fe102cc3de\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.512395 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.512334    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:06.542685 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.542661    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:06.785592 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.785569    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"]
Nov 03 08:02:06.787412 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:02:06.787386    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9ad9ff_3d0f_4994_829c_81fe102cc3de.slice/crio-3f356b3679bd9947bfaeb6db1496bdb43d3fff2da6e10f1a9cb701dc917dbc8b WatchSource:0}: Error finding container 3f356b3679bd9947bfaeb6db1496bdb43d3fff2da6e10f1a9cb701dc917dbc8b: Status 404 returned error can't find the container with id 3f356b3679bd9947bfaeb6db1496bdb43d3fff2da6e10f1a9cb701dc917dbc8b
Nov 03 08:02:06.790453 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.790418    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:02:06.790554 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.790487    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:02:06.790554 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.790520    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:02:06.824874 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.824851    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"]
Nov 03 08:02:06.825901 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:02:06.825876    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337a39e3_b285_4d8c_aa1e_210a6e46763a.slice/crio-b330ad09eed9c381494780ec2692b2b4579f05c627a9401d69240cff675095a8 WatchSource:0}: Error finding container b330ad09eed9c381494780ec2692b2b4579f05c627a9401d69240cff675095a8: Status 404 returned error can't find the container with id b330ad09eed9c381494780ec2692b2b4579f05c627a9401d69240cff675095a8
Nov 03 08:02:06.827907 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.827876    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj" event={"ID":"0c9ad9ff-3d0f-4994-829c-81fe102cc3de","Type":"ContainerStarted","Data":"3f356b3679bd9947bfaeb6db1496bdb43d3fff2da6e10f1a9cb701dc917dbc8b"}
Nov 03 08:02:06.829106 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.829077    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:02:06.829177 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.829153    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:02:06.829216 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:06.829183    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:02:07.834195 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:07.834161    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj" event={"ID":"0c9ad9ff-3d0f-4994-829c-81fe102cc3de","Type":"ContainerStarted","Data":"4c2db8ebb68aad71d221ce925e6420fd3769b7c131387ba914528479abbb2b93"}
Nov 03 08:02:07.836210 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:07.836187    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz" event={"ID":"337a39e3-b285-4d8c-aa1e-210a6e46763a","Type":"ContainerStarted","Data":"6bcc3636f20cc60c05dc615e934d95113699f8600830fd3ced73a7671d87d9b7"}
Nov 03 08:02:07.836287 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:07.836231    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz" event={"ID":"337a39e3-b285-4d8c-aa1e-210a6e46763a","Type":"ContainerStarted","Data":"b330ad09eed9c381494780ec2692b2b4579f05c627a9401d69240cff675095a8"}
Nov 03 08:02:07.852245 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:07.852199    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj" podStartSLOduration=1.8521867319999998 podStartE2EDuration="1.852186732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:07.850273301 +0000 UTC m=+525.964737230" watchObservedRunningTime="2026-11-03 08:02:07.852186732 +0000 UTC m=+525.966650662"
Nov 03 08:02:07.879782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:07.879063    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz" podStartSLOduration=1.87904363 podStartE2EDuration="1.87904363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:07.878076989 +0000 UTC m=+525.992540920" watchObservedRunningTime="2026-11-03 08:02:07.87904363 +0000 UTC m=+525.993507562"
Nov 03 08:02:08.512743 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:08.512712    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:08.513604 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:08.513582    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-87snj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 08:02:08.513715 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:08.513621    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj" podUID="0c9ad9ff-3d0f-4994-829c-81fe102cc3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 08:02:08.542988 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:08.542964    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:08.543903 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:08.543887    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tlbnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 08:02:08.543984 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:08.543927    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz" podUID="337a39e3-b285-4d8c-aa1e-210a6e46763a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 08:02:09.513379 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:09.513338    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-87snj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 08:02:09.513741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:09.513400    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj" podUID="0c9ad9ff-3d0f-4994-829c-81fe102cc3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 08:02:09.543023 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:09.542994    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tlbnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 08:02:09.543161 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:09.543036    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz" podUID="337a39e3-b285-4d8c-aa1e-210a6e46763a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 08:02:10.513414 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:10.513374    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-87snj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 08:02:10.513916 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:10.513433    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj" podUID="0c9ad9ff-3d0f-4994-829c-81fe102cc3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 08:02:10.543036 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:10.542992    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tlbnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 08:02:10.543167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:10.543062    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz" podUID="337a39e3-b285-4d8c-aa1e-210a6e46763a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 08:02:11.513238 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:11.513200    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-87snj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 08:02:11.513426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:11.513269    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj" podUID="0c9ad9ff-3d0f-4994-829c-81fe102cc3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 08:02:11.543226 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:11.543192    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tlbnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 08:02:11.543585 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:11.543239    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz" podUID="337a39e3-b285-4d8c-aa1e-210a6e46763a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 08:02:12.513258 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:12.513218    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-87snj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 08:02:12.513436 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:12.513279    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj" podUID="0c9ad9ff-3d0f-4994-829c-81fe102cc3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 08:02:12.543152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:12.543123    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tlbnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 08:02:12.543231 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:12.543173    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz" podUID="337a39e3-b285-4d8c-aa1e-210a6e46763a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 08:02:13.513407 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:13.513375    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-87snj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Nov 03 08:02:13.513806 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:13.513427    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj" podUID="0c9ad9ff-3d0f-4994-829c-81fe102cc3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Nov 03 08:02:13.543729 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:13.543700    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tlbnz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Nov 03 08:02:13.543892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:13.543749    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz" podUID="337a39e3-b285-4d8c-aa1e-210a6e46763a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Nov 03 08:02:15.078778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:15.078727    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:15.079719 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:15.079693    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:15.080379 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:15.080353    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj"
Nov 03 08:02:15.124917 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:15.124887    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:15.125456 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:15.125430    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:15.126560 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:15.126532    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:02:22.050233 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:02:22.050196    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:02:45.189210 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.189172    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"]
Nov 03 08:02:45.193604 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.193581    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.203036 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.203009    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"]
Nov 03 08:02:45.212854 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.212829    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-bd085d-openshift-default-dockercfg-v65xr\""
Nov 03 08:02:45.343638 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.343595    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.343822 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.343647    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q42g5\" (UniqueName: \"kubernetes.io/projected/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-kube-api-access-q42g5\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.343822 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.343693    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.343822 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.343724    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.343822 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.343753    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.344093 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.343879    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.344093 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.343930    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.344093 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.343984    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.344093 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.344038    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.429726 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.429692    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"]
Nov 03 08:02:45.434329 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.434302    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.447011 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.446985    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.447667 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.447639    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.447811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.447691    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.448406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.448382    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.448547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.448528    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.448674 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.448654    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"]
Nov 03 08:02:45.448826 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.448805    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.449813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.448888    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.449907 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.449869    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q42g5\" (UniqueName: \"kubernetes.io/projected/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-kube-api-access-q42g5\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.449964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.449915    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.449964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.449935    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.449964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.449954    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.450097 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.450045    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.451743 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.450294    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.451743 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.451696    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.452063 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.452038    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.453370 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.453346    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.461904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.461879    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q42g5\" (UniqueName: \"kubernetes.io/projected/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-kube-api-access-q42g5\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.462376 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.462356    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.471896 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.471670    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-15a633-openshift-default-dockercfg-vdsjd\""
Nov 03 08:02:45.514537 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.514317    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:45.514999 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.514977    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"]
Nov 03 08:02:45.518450 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.518430    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.532156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.532133    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"]
Nov 03 08:02:45.549923 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.549900    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-19bb0a-openshift-default-dockercfg-2phtd\""
Nov 03 08:02:45.551584 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.551548    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.551664 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.551626    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.551717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.551665    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.551717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.551712    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwfbg\" (UniqueName: \"kubernetes.io/projected/4f3b3503-699a-4697-b121-351ad81bad8a-kube-api-access-qwfbg\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.551853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.551749    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3b3503-699a-4697-b121-351ad81bad8a-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.551853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.551800    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3b3503-699a-4697-b121-351ad81bad8a-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.551955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.551909    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.552007 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.551972    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3b3503-699a-4697-b121-351ad81bad8a-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.552059 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.552006    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.652785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.652720    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3b3503-699a-4697-b121-351ad81bad8a-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.652991 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.652806    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.652991 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.652888    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.652991 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.652927    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.653165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.653058    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.653165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.653095    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.653165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.653156    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwfbg\" (UniqueName: \"kubernetes.io/projected/4f3b3503-699a-4697-b121-351ad81bad8a-kube-api-access-qwfbg\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.653324 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.653195    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3b3503-699a-4697-b121-351ad81bad8a-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.653324 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.653229    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks9ln\" (UniqueName: \"kubernetes.io/projected/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-kube-api-access-ks9ln\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.653324 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.653257    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3b3503-699a-4697-b121-351ad81bad8a-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.653324 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.653319    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.653526 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.653364    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.653526 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.653402    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.653526 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.653430    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.653526 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.653460    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.653526 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.653469    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.653526 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.653490    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.653526 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.653495    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.653997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.653562    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.653997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.653717    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.654106 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.654024    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.654161 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.654124    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3b3503-699a-4697-b121-351ad81bad8a-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.654521 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.654474    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.656317 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.656295    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.656461 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.656410    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3b3503-699a-4697-b121-351ad81bad8a-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.665935 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.665907    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3b3503-699a-4697-b121-351ad81bad8a-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.670332 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.670308    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwfbg\" (UniqueName: \"kubernetes.io/projected/4f3b3503-699a-4697-b121-351ad81bad8a-kube-api-access-qwfbg\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.755426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.755317    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.755426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.755388    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.755689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.755427    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.755689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.755533    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.755689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.755567    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.755689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.755597    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.755689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.755631    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.755689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.755683    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.756109 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.755852    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks9ln\" (UniqueName: \"kubernetes.io/projected/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-kube-api-access-ks9ln\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.756665 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.756628    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.757544 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.757506    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.757888 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.757798    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.757888 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.757851    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.758219 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.758200    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.759046 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.759021    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.759864 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.759840    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:45.760641 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.760615    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.777142 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.777115    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks9ln\" (UniqueName: \"kubernetes.io/projected/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-kube-api-access-ks9ln\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.782262 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.782235    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:45.812176 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.812142    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"]
Nov 03 08:02:45.812566 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:02:45.812541    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc257d6_34f4_412a_80c4_9a7d4462d2f9.slice/crio-b67f45a4d154e302a94f306d0d1e175cb90876a43b232abaf52926f3edae9a04 WatchSource:0}: Error finding container b67f45a4d154e302a94f306d0d1e175cb90876a43b232abaf52926f3edae9a04: Status 404 returned error can't find the container with id b67f45a4d154e302a94f306d0d1e175cb90876a43b232abaf52926f3edae9a04
Nov 03 08:02:45.814812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.814747    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:02:45.814914 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.814856    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:02:45.814914 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.814901    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:02:45.866204 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:45.866182    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:46.037790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:46.037700    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn" event={"ID":"4fc257d6-34f4-412a-80c4-9a7d4462d2f9","Type":"ContainerStarted","Data":"ac1ef05fa603dd69fa4d8d1d16e234f61492af355d24860a2e6e264a6cb2ed44"}
Nov 03 08:02:46.037790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:46.037774    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn" event={"ID":"4fc257d6-34f4-412a-80c4-9a7d4462d2f9","Type":"ContainerStarted","Data":"b67f45a4d154e302a94f306d0d1e175cb90876a43b232abaf52926f3edae9a04"}
Nov 03 08:02:46.053506 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:46.053480    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"]
Nov 03 08:02:46.055288 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:02:46.055262    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f3b3503_699a_4697_b121_351ad81bad8a.slice/crio-89885ad42ff98109be87400735033af848932d78a9e326115506a6914e817e2a WatchSource:0}: Error finding container 89885ad42ff98109be87400735033af848932d78a9e326115506a6914e817e2a: Status 404 returned error can't find the container with id 89885ad42ff98109be87400735033af848932d78a9e326115506a6914e817e2a
Nov 03 08:02:46.058092 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:46.058052    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:02:46.058210 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:46.058128    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:02:46.058210 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:46.058163    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:02:46.059807 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:46.059750    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn" podStartSLOduration=1.059738507 podStartE2EDuration="1.059738507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:46.057362421 +0000 UTC m=+564.171826352" watchObservedRunningTime="2026-11-03 08:02:46.059738507 +0000 UTC m=+564.174202437"
Nov 03 08:02:46.149634 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:46.149611    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"]
Nov 03 08:02:46.152385 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:02:46.152357    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0fa1f2b_15c4_48a8_9eef_d87bbcc6aafc.slice/crio-213f6107d343f3c6243d27a919654489b0b46e70bde2aac52386cdf099109372 WatchSource:0}: Error finding container 213f6107d343f3c6243d27a919654489b0b46e70bde2aac52386cdf099109372: Status 404 returned error can't find the container with id 213f6107d343f3c6243d27a919654489b0b46e70bde2aac52386cdf099109372
Nov 03 08:02:46.154988 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:46.154946    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:02:46.155076 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:46.155037    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:02:46.155116 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:46.155078    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:02:46.515568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:46.515532    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:46.520127 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:46.520106    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:47.051334 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:47.051298    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt" event={"ID":"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc","Type":"ContainerStarted","Data":"dee6741d7474f0befcf7b11602ae9c3b830feec2b33134e736cb84c8c8cac57e"}
Nov 03 08:02:47.051547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:47.051394    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt" event={"ID":"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc","Type":"ContainerStarted","Data":"213f6107d343f3c6243d27a919654489b0b46e70bde2aac52386cdf099109372"}
Nov 03 08:02:47.053291 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:47.053263    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p" event={"ID":"4f3b3503-699a-4697-b121-351ad81bad8a","Type":"ContainerStarted","Data":"776e6bbde0a6b87bf9d2c28e2d8a298aeabc01552cdab214a0425efce769f8f3"}
Nov 03 08:02:47.053417 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:47.053333    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:47.053417 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:47.053354    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p" event={"ID":"4f3b3503-699a-4697-b121-351ad81bad8a","Type":"ContainerStarted","Data":"89885ad42ff98109be87400735033af848932d78a9e326115506a6914e817e2a"}
Nov 03 08:02:47.054445 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:47.054425    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:02:47.071379 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:47.071329    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt" podStartSLOduration=2.071316576 podStartE2EDuration="2.071316576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:47.069216515 +0000 UTC m=+565.183680447" watchObservedRunningTime="2026-11-03 08:02:47.071316576 +0000 UTC m=+565.185780506"
Nov 03 08:02:47.105584 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:47.105527    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p" podStartSLOduration=2.105511987 podStartE2EDuration="2.105511987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:47.103053699 +0000 UTC m=+565.217517633" watchObservedRunningTime="2026-11-03 08:02:47.105511987 +0000 UTC m=+565.219975918"
Nov 03 08:02:47.760821 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:47.760785    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:47.765477 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:47.765455    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:47.867405 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:47.867374    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:47.872265 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:47.872243    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:48.058955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:48.058873    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:48.058955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:48.058932    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:48.059630 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:48.059605    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:02:48.059703 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:48.059659    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:02:50.812293 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:50.812258    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44"]
Nov 03 08:02:50.816646 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:50.816624    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44"
Nov 03 08:02:50.837521 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:50.837493    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44"]
Nov 03 08:02:50.843439 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:50.843398    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-6bc553-dockercfg-9j9k5\""
Nov 03 08:02:50.900608 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:50.900580    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-6bc553\""
Nov 03 08:02:50.906945 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:50.906923    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d84pm\" (UniqueName: \"kubernetes.io/projected/56258879-0d2c-457c-a0d2-0e7c378bed50-kube-api-access-d84pm\") pod \"payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44\" (UID: \"56258879-0d2c-457c-a0d2-0e7c378bed50\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44"
Nov 03 08:02:50.907025 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:50.906981    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56258879-0d2c-457c-a0d2-0e7c378bed50-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44\" (UID: \"56258879-0d2c-457c-a0d2-0e7c378bed50\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44"
Nov 03 08:02:51.008646 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:51.008604    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d84pm\" (UniqueName: \"kubernetes.io/projected/56258879-0d2c-457c-a0d2-0e7c378bed50-kube-api-access-d84pm\") pod \"payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44\" (UID: \"56258879-0d2c-457c-a0d2-0e7c378bed50\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44"
Nov 03 08:02:51.008835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:51.008693    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56258879-0d2c-457c-a0d2-0e7c378bed50-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44\" (UID: \"56258879-0d2c-457c-a0d2-0e7c378bed50\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44"
Nov 03 08:02:51.009421 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:51.009400    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56258879-0d2c-457c-a0d2-0e7c378bed50-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44\" (UID: \"56258879-0d2c-457c-a0d2-0e7c378bed50\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44"
Nov 03 08:02:51.016401 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:51.016375    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d84pm\" (UniqueName: \"kubernetes.io/projected/56258879-0d2c-457c-a0d2-0e7c378bed50-kube-api-access-d84pm\") pod \"payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44\" (UID: \"56258879-0d2c-457c-a0d2-0e7c378bed50\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44"
Nov 03 08:02:51.145942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:51.145911    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44"
Nov 03 08:02:51.182867 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:51.182818    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd"]
Nov 03 08:02:51.187873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:51.187854    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd"
Nov 03 08:02:51.197944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:51.197866    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-3e88b7\""
Nov 03 08:02:51.202462 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:51.202390    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd"]
Nov 03 08:02:51.210960 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:51.210249    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb961482-e401-47f5-8aff-f604b364d631-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd\" (UID: \"cb961482-e401-47f5-8aff-f604b364d631\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd"
Nov 03 08:02:51.210960 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:51.210425    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpwmr\" (UniqueName: \"kubernetes.io/projected/cb961482-e401-47f5-8aff-f604b364d631-kube-api-access-cpwmr\") pod \"payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd\" (UID: \"cb961482-e401-47f5-8aff-f604b364d631\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd"
Nov 03 08:02:51.312692 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:51.312570    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb961482-e401-47f5-8aff-f604b364d631-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd\" (UID: \"cb961482-e401-47f5-8aff-f604b364d631\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd"
Nov 03 08:02:51.313569 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:51.313521    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb961482-e401-47f5-8aff-f604b364d631-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd\" (UID: \"cb961482-e401-47f5-8aff-f604b364d631\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd"
Nov 03 08:02:51.314852 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:51.314442    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpwmr\" (UniqueName: \"kubernetes.io/projected/cb961482-e401-47f5-8aff-f604b364d631-kube-api-access-cpwmr\") pod \"payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd\" (UID: \"cb961482-e401-47f5-8aff-f604b364d631\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd"
Nov 03 08:02:51.324490 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:51.324461    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpwmr\" (UniqueName: \"kubernetes.io/projected/cb961482-e401-47f5-8aff-f604b364d631-kube-api-access-cpwmr\") pod \"payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd\" (UID: \"cb961482-e401-47f5-8aff-f604b364d631\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd"
Nov 03 08:02:51.428994 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:51.428962    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44"]
Nov 03 08:02:51.431069 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:02:51.431041    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56258879_0d2c_457c_a0d2_0e7c378bed50.slice/crio-21ebfdd1ab12eeb5a4e922df261e64ed44a9fccffa455feb48c04fd22529d643 WatchSource:0}: Error finding container 21ebfdd1ab12eeb5a4e922df261e64ed44a9fccffa455feb48c04fd22529d643: Status 404 returned error can't find the container with id 21ebfdd1ab12eeb5a4e922df261e64ed44a9fccffa455feb48c04fd22529d643
Nov 03 08:02:51.509234 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:51.509213    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd"
Nov 03 08:02:51.791648 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:51.791624    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd"]
Nov 03 08:02:51.793504 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:02:51.793478    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb961482_e401_47f5_8aff_f604b364d631.slice/crio-612729ff76eca47f2a71e185ed5b5739c2e9e17e59b380030478d57b2852522e WatchSource:0}: Error finding container 612729ff76eca47f2a71e185ed5b5739c2e9e17e59b380030478d57b2852522e: Status 404 returned error can't find the container with id 612729ff76eca47f2a71e185ed5b5739c2e9e17e59b380030478d57b2852522e
Nov 03 08:02:52.067995 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:02:52.067909    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:02:52.075933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.075902    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9"]
Nov 03 08:02:52.080186 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.080167    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9"
Nov 03 08:02:52.084748 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.084717    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd" event={"ID":"cb961482-e401-47f5-8aff-f604b364d631","Type":"ContainerStarted","Data":"612729ff76eca47f2a71e185ed5b5739c2e9e17e59b380030478d57b2852522e"}
Nov 03 08:02:52.088145 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.088088    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44" event={"ID":"56258879-0d2c-457c-a0d2-0e7c378bed50","Type":"ContainerStarted","Data":"21ebfdd1ab12eeb5a4e922df261e64ed44a9fccffa455feb48c04fd22529d643"}
Nov 03 08:02:52.089585 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.089562    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9"]
Nov 03 08:02:52.123985 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.123941    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/917243d5-933a-4f61-9e76-72ec95f6a0dd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9\" (UID: \"917243d5-933a-4f61-9e76-72ec95f6a0dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9"
Nov 03 08:02:52.124875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.124092    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w68bt\" (UniqueName: \"kubernetes.io/projected/917243d5-933a-4f61-9e76-72ec95f6a0dd-kube-api-access-w68bt\") pod \"payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9\" (UID: \"917243d5-933a-4f61-9e76-72ec95f6a0dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9"
Nov 03 08:02:52.164270 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.164137    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-ffe7a8-dockercfg-qd5z5\""
Nov 03 08:02:52.164270 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.164238    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-ffe7a8\""
Nov 03 08:02:52.225586 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.225555    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w68bt\" (UniqueName: \"kubernetes.io/projected/917243d5-933a-4f61-9e76-72ec95f6a0dd-kube-api-access-w68bt\") pod \"payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9\" (UID: \"917243d5-933a-4f61-9e76-72ec95f6a0dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9"
Nov 03 08:02:52.225805 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.225624    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/917243d5-933a-4f61-9e76-72ec95f6a0dd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9\" (UID: \"917243d5-933a-4f61-9e76-72ec95f6a0dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9"
Nov 03 08:02:52.226420 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.226390    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/917243d5-933a-4f61-9e76-72ec95f6a0dd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9\" (UID: \"917243d5-933a-4f61-9e76-72ec95f6a0dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9"
Nov 03 08:02:52.232658 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.232637    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w68bt\" (UniqueName: \"kubernetes.io/projected/917243d5-933a-4f61-9e76-72ec95f6a0dd-kube-api-access-w68bt\") pod \"payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9\" (UID: \"917243d5-933a-4f61-9e76-72ec95f6a0dd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9"
Nov 03 08:02:52.397976 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.397948    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9"
Nov 03 08:02:52.727459 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.727374    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9"]
Nov 03 08:02:52.731546 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:02:52.731521    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod917243d5_933a_4f61_9e76_72ec95f6a0dd.slice/crio-3f15a0b850a9e3084be6c54735290d40b7c3c81e19a90304249fb17d35ed973a WatchSource:0}: Error finding container 3f15a0b850a9e3084be6c54735290d40b7c3c81e19a90304249fb17d35ed973a: Status 404 returned error can't find the container with id 3f15a0b850a9e3084be6c54735290d40b7c3c81e19a90304249fb17d35ed973a
Nov 03 08:02:52.816095 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.816052    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn"]
Nov 03 08:02:52.820323 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.820298    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn"
Nov 03 08:02:52.825040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.825014    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-76ef13\""
Nov 03 08:02:52.825183 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.825165    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-76ef13-dockercfg-sghvt\""
Nov 03 08:02:52.828324 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.828227    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn"]
Nov 03 08:02:52.933506 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.933410    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/063e3656-6675-4336-a3c1-cf96b568be28-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn\" (UID: \"063e3656-6675-4336-a3c1-cf96b568be28\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn"
Nov 03 08:02:52.933506 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:52.933460    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhr4p\" (UniqueName: \"kubernetes.io/projected/063e3656-6675-4336-a3c1-cf96b568be28-kube-api-access-bhr4p\") pod \"payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn\" (UID: \"063e3656-6675-4336-a3c1-cf96b568be28\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn"
Nov 03 08:02:53.035061 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.034986    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/063e3656-6675-4336-a3c1-cf96b568be28-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn\" (UID: \"063e3656-6675-4336-a3c1-cf96b568be28\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn"
Nov 03 08:02:53.035061 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.035025    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhr4p\" (UniqueName: \"kubernetes.io/projected/063e3656-6675-4336-a3c1-cf96b568be28-kube-api-access-bhr4p\") pod \"payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn\" (UID: \"063e3656-6675-4336-a3c1-cf96b568be28\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn"
Nov 03 08:02:53.035930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.035901    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/063e3656-6675-4336-a3c1-cf96b568be28-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn\" (UID: \"063e3656-6675-4336-a3c1-cf96b568be28\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn"
Nov 03 08:02:53.042593 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.042567    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhr4p\" (UniqueName: \"kubernetes.io/projected/063e3656-6675-4336-a3c1-cf96b568be28-kube-api-access-bhr4p\") pod \"payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn\" (UID: \"063e3656-6675-4336-a3c1-cf96b568be28\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn"
Nov 03 08:02:53.098373 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.098337    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9" event={"ID":"917243d5-933a-4f61-9e76-72ec95f6a0dd","Type":"ContainerStarted","Data":"3f15a0b850a9e3084be6c54735290d40b7c3c81e19a90304249fb17d35ed973a"}
Nov 03 08:02:53.148766 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.148730    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn"
Nov 03 08:02:53.277481 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.277449    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb"]
Nov 03 08:02:53.282838 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.282811    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb"
Nov 03 08:02:53.290825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.288848    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb"]
Nov 03 08:02:53.290825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.290278    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-15a633-dockercfg-25sdt\""
Nov 03 08:02:53.291066 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.291046    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-15a633\""
Nov 03 08:02:53.326562 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.326536    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m"]
Nov 03 08:02:53.332777 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.332737    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m"
Nov 03 08:02:53.339013 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.338990    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d449195e-0ed5-4c8e-ac1b-3242198699f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb\" (UID: \"d449195e-0ed5-4c8e-ac1b-3242198699f7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb"
Nov 03 08:02:53.339139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.339042    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfsd5\" (UniqueName: \"kubernetes.io/projected/d449195e-0ed5-4c8e-ac1b-3242198699f7-kube-api-access-mfsd5\") pod \"payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb\" (UID: \"d449195e-0ed5-4c8e-ac1b-3242198699f7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb"
Nov 03 08:02:53.339139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.339127    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m"]
Nov 03 08:02:53.439815 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.439775    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d449195e-0ed5-4c8e-ac1b-3242198699f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb\" (UID: \"d449195e-0ed5-4c8e-ac1b-3242198699f7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb"
Nov 03 08:02:53.439990 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.439851    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfsd5\" (UniqueName: \"kubernetes.io/projected/d449195e-0ed5-4c8e-ac1b-3242198699f7-kube-api-access-mfsd5\") pod \"payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb\" (UID: \"d449195e-0ed5-4c8e-ac1b-3242198699f7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb"
Nov 03 08:02:53.439990 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.439887    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdhvz\" (UniqueName: \"kubernetes.io/projected/cb3722fc-983b-4b33-9269-8e559475aff5-kube-api-access-kdhvz\") pod \"payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m\" (UID: \"cb3722fc-983b-4b33-9269-8e559475aff5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m"
Nov 03 08:02:53.439990 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.439951    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3722fc-983b-4b33-9269-8e559475aff5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m\" (UID: \"cb3722fc-983b-4b33-9269-8e559475aff5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m"
Nov 03 08:02:53.440637 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.440607    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d449195e-0ed5-4c8e-ac1b-3242198699f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb\" (UID: \"d449195e-0ed5-4c8e-ac1b-3242198699f7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb"
Nov 03 08:02:53.447629 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.447602    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfsd5\" (UniqueName: \"kubernetes.io/projected/d449195e-0ed5-4c8e-ac1b-3242198699f7-kube-api-access-mfsd5\") pod \"payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb\" (UID: \"d449195e-0ed5-4c8e-ac1b-3242198699f7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb"
Nov 03 08:02:53.452098 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.452069    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn"]
Nov 03 08:02:53.480099 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:02:53.480070    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod063e3656_6675_4336_a3c1_cf96b568be28.slice/crio-c367495c8ef0d05930237ef4df513308e0bb2ed1ef76363bdd28ac0ca9d31916 WatchSource:0}: Error finding container c367495c8ef0d05930237ef4df513308e0bb2ed1ef76363bdd28ac0ca9d31916: Status 404 returned error can't find the container with id c367495c8ef0d05930237ef4df513308e0bb2ed1ef76363bdd28ac0ca9d31916
Nov 03 08:02:53.540891 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.540828    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdhvz\" (UniqueName: \"kubernetes.io/projected/cb3722fc-983b-4b33-9269-8e559475aff5-kube-api-access-kdhvz\") pod \"payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m\" (UID: \"cb3722fc-983b-4b33-9269-8e559475aff5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m"
Nov 03 08:02:53.541028 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.540914    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3722fc-983b-4b33-9269-8e559475aff5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m\" (UID: \"cb3722fc-983b-4b33-9269-8e559475aff5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m"
Nov 03 08:02:53.541822 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.541793    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3722fc-983b-4b33-9269-8e559475aff5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m\" (UID: \"cb3722fc-983b-4b33-9269-8e559475aff5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m"
Nov 03 08:02:53.549546 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.549523    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdhvz\" (UniqueName: \"kubernetes.io/projected/cb3722fc-983b-4b33-9269-8e559475aff5-kube-api-access-kdhvz\") pod \"payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m\" (UID: \"cb3722fc-983b-4b33-9269-8e559475aff5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m"
Nov 03 08:02:53.604271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.604238    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb"
Nov 03 08:02:53.654391 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.654361    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m"
Nov 03 08:02:53.918576 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.918494    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb"]
Nov 03 08:02:53.923136 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:02:53.923094    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd449195e_0ed5_4c8e_ac1b_3242198699f7.slice/crio-0782d2243d4471d30e8e640e4f04c705da68325525558dc22ab9be2eab1dddf7 WatchSource:0}: Error finding container 0782d2243d4471d30e8e640e4f04c705da68325525558dc22ab9be2eab1dddf7: Status 404 returned error can't find the container with id 0782d2243d4471d30e8e640e4f04c705da68325525558dc22ab9be2eab1dddf7
Nov 03 08:02:53.969217 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.969188    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m"]
Nov 03 08:02:53.970315 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:02:53.970289    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3722fc_983b_4b33_9269_8e559475aff5.slice/crio-9fc9fdb6f59d740369d9e95debabbde69b90a586d2224f5d4c383706f48ed2c7 WatchSource:0}: Error finding container 9fc9fdb6f59d740369d9e95debabbde69b90a586d2224f5d4c383706f48ed2c7: Status 404 returned error can't find the container with id 9fc9fdb6f59d740369d9e95debabbde69b90a586d2224f5d4c383706f48ed2c7
Nov 03 08:02:53.971772 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.971731    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg"]
Nov 03 08:02:53.976880 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.976845    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg"
Nov 03 08:02:53.997586 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.997564    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-19bb0a-dockercfg-64t7v\""
Nov 03 08:02:53.998975 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:53.998952    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg"]
Nov 03 08:02:54.059852 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:54.058518    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2bafc12-e2f3-46ee-8f98-01e46c2c2787-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg\" (UID: \"a2bafc12-e2f3-46ee-8f98-01e46c2c2787\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg"
Nov 03 08:02:54.059852 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:54.058592    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n7sr\" (UniqueName: \"kubernetes.io/projected/a2bafc12-e2f3-46ee-8f98-01e46c2c2787-kube-api-access-7n7sr\") pod \"payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg\" (UID: \"a2bafc12-e2f3-46ee-8f98-01e46c2c2787\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg"
Nov 03 08:02:54.062936 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:54.062859    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-19bb0a\""
Nov 03 08:02:54.114737 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:54.114634    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb" event={"ID":"d449195e-0ed5-4c8e-ac1b-3242198699f7","Type":"ContainerStarted","Data":"0782d2243d4471d30e8e640e4f04c705da68325525558dc22ab9be2eab1dddf7"}
Nov 03 08:02:54.117287 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:54.117225    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m" event={"ID":"cb3722fc-983b-4b33-9269-8e559475aff5","Type":"ContainerStarted","Data":"9fc9fdb6f59d740369d9e95debabbde69b90a586d2224f5d4c383706f48ed2c7"}
Nov 03 08:02:54.121954 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:54.121912    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn" event={"ID":"063e3656-6675-4336-a3c1-cf96b568be28","Type":"ContainerStarted","Data":"c367495c8ef0d05930237ef4df513308e0bb2ed1ef76363bdd28ac0ca9d31916"}
Nov 03 08:02:54.160214 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:54.160179    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2bafc12-e2f3-46ee-8f98-01e46c2c2787-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg\" (UID: \"a2bafc12-e2f3-46ee-8f98-01e46c2c2787\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg"
Nov 03 08:02:54.160367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:54.160247    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n7sr\" (UniqueName: \"kubernetes.io/projected/a2bafc12-e2f3-46ee-8f98-01e46c2c2787-kube-api-access-7n7sr\") pod \"payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg\" (UID: \"a2bafc12-e2f3-46ee-8f98-01e46c2c2787\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg"
Nov 03 08:02:54.161065 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:54.161041    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2bafc12-e2f3-46ee-8f98-01e46c2c2787-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg\" (UID: \"a2bafc12-e2f3-46ee-8f98-01e46c2c2787\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg"
Nov 03 08:02:54.171846 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:54.168727    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n7sr\" (UniqueName: \"kubernetes.io/projected/a2bafc12-e2f3-46ee-8f98-01e46c2c2787-kube-api-access-7n7sr\") pod \"payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg\" (UID: \"a2bafc12-e2f3-46ee-8f98-01e46c2c2787\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg"
Nov 03 08:02:54.281116 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:54.281071    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:02:54.298365 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:54.298145    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg"
Nov 03 08:02:54.679083 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:54.678993    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg"]
Nov 03 08:02:55.140263 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:55.139937    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg" event={"ID":"a2bafc12-e2f3-46ee-8f98-01e46c2c2787","Type":"ContainerStarted","Data":"cd2d5b6a6020f6fb19771efe29f05a5c406c6d58222f86f6349bab46ccab1589"}
Nov 03 08:02:56.165372 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:56.164841    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44" event={"ID":"56258879-0d2c-457c-a0d2-0e7c378bed50","Type":"ContainerStarted","Data":"286a57fd568b2648bb4dd3481b47fb5a2a051c7afc381e04413c0db92e6d2d73"}
Nov 03 08:02:56.165372 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:56.165046    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44"
Nov 03 08:02:56.181071 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:56.181026    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44" podStartSLOduration=1.933164615 podStartE2EDuration="6.181013341s" totalImagesPullingTime="4.247848726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:56.178436648 +0000 UTC m=+574.292900581" watchObservedRunningTime="2026-11-03 08:02:56.181013341 +0000 UTC m=+574.295477269"
Nov 03 08:02:57.172258 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:57.172190    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9" event={"ID":"917243d5-933a-4f61-9e76-72ec95f6a0dd","Type":"ContainerStarted","Data":"311bea97dce6e955643a8cacfee6d4af6170e9aa483d13c65db7926ed4bd1e30"}
Nov 03 08:02:57.172638 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:57.172262    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9"
Nov 03 08:02:57.173742 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:57.173718    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m" event={"ID":"cb3722fc-983b-4b33-9269-8e559475aff5","Type":"ContainerStarted","Data":"84944f5477498a6baef9b706c16bad14f7b62a33ae6c8ab56da34661d69850c2"}
Nov 03 08:02:57.173853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:57.173804    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m"
Nov 03 08:02:57.175306 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:57.175283    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn" event={"ID":"063e3656-6675-4336-a3c1-cf96b568be28","Type":"ContainerStarted","Data":"89d6a74919c8b739e8ca254ff8e9552e65775ceab1360487a2b6a05bb8333217"}
Nov 03 08:02:57.175383 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:57.175366    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn"
Nov 03 08:02:57.176892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:57.176868    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd" event={"ID":"cb961482-e401-47f5-8aff-f604b364d631","Type":"ContainerStarted","Data":"493c820308728c3e98ad907864a52d22363893328c3b730567fe658c759f2a07"}
Nov 03 08:02:57.176961 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:57.176917    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd"
Nov 03 08:02:57.178328 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:57.178306    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg" event={"ID":"a2bafc12-e2f3-46ee-8f98-01e46c2c2787","Type":"ContainerStarted","Data":"9a349cac83afa17a6fd7dcd2183a4c8615aecc6cdb9c5ff2a4897155383bcb1f"}
Nov 03 08:02:57.178419 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:57.178352    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg"
Nov 03 08:02:57.179753 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:57.179734    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb" event={"ID":"d449195e-0ed5-4c8e-ac1b-3242198699f7","Type":"ContainerStarted","Data":"5af597feaa1c3de87495549902e7b6a22b74cecb6a7d6392963fd11a76083a22"}
Nov 03 08:02:57.179924 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:57.179892    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb"
Nov 03 08:02:57.188016 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:57.187969    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9" podStartSLOduration=0.972682813 podStartE2EDuration="5.187953243s" totalImagesPullingTime="4.21527043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:57.186648427 +0000 UTC m=+575.301112358" watchObservedRunningTime="2026-11-03 08:02:57.187953243 +0000 UTC m=+575.302417178"
Nov 03 08:02:57.204262 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:57.204221    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m" podStartSLOduration=1.5992098590000001 podStartE2EDuration="4.204209884s" totalImagesPullingTime="2.605000025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:57.200957839 +0000 UTC m=+575.315421770" watchObservedRunningTime="2026-11-03 08:02:57.204209884 +0000 UTC m=+575.318673815"
Nov 03 08:02:57.217003 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:57.216963    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn" podStartSLOduration=2.473569066 podStartE2EDuration="5.216951955s" totalImagesPullingTime="2.743382889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:57.214208192 +0000 UTC m=+575.328672138" watchObservedRunningTime="2026-11-03 08:02:57.216951955 +0000 UTC m=+575.331415886"
Nov 03 08:02:57.228794 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:57.228735    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg" podStartSLOduration=2.063378017 podStartE2EDuration="4.228725655s" totalImagesPullingTime="2.165347638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:57.228463798 +0000 UTC m=+575.342927730" watchObservedRunningTime="2026-11-03 08:02:57.228725655 +0000 UTC m=+575.343189585"
Nov 03 08:02:57.245809 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:57.245751    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd" podStartSLOduration=1.31133995 podStartE2EDuration="6.245739712s" totalImagesPullingTime="4.934399762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:57.244642778 +0000 UTC m=+575.359106708" watchObservedRunningTime="2026-11-03 08:02:57.245739712 +0000 UTC m=+575.360203669"
Nov 03 08:02:57.271160 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:57.270993    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb" podStartSLOduration=1.575943557 podStartE2EDuration="4.270975788s" totalImagesPullingTime="2.695032231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:02:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:02:57.269185118 +0000 UTC m=+575.383649050" watchObservedRunningTime="2026-11-03 08:02:57.270975788 +0000 UTC m=+575.385439724"
Nov 03 08:02:59.828379 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:02:59.828347    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:03:07.188714 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:07.188685    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44"
Nov 03 08:03:08.190095 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:08.189204    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg"
Nov 03 08:03:08.190095 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:08.190002    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd"
Nov 03 08:03:08.191624 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:08.190982    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn"
Nov 03 08:03:08.193375 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:08.193292    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb"
Nov 03 08:03:08.193491 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:08.193404    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9"
Nov 03 08:03:08.193649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:08.193633    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m"
Nov 03 08:03:10.718413 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:10.718357    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:03:11.325126 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:11.325082    5070 generic.go:356] "Generic (PLEG): container finished" podID="f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" containerID="dee6741d7474f0befcf7b11602ae9c3b830feec2b33134e736cb84c8c8cac57e" exitCode=0
Nov 03 08:03:11.325447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:11.325289    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt" event={"ID":"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc","Type":"ContainerDied","Data":"dee6741d7474f0befcf7b11602ae9c3b830feec2b33134e736cb84c8c8cac57e"}
Nov 03 08:03:11.325798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:11.325751    5070 scope.go:122] "RemoveContainer" containerID="dee6741d7474f0befcf7b11602ae9c3b830feec2b33134e736cb84c8c8cac57e"
Nov 03 08:03:11.327294 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:11.327258    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:03:11.327395 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:11.327334    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:03:11.327395 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:11.327376    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:03:12.333316 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.333278    5070 generic.go:356] "Generic (PLEG): container finished" podID="4fc257d6-34f4-412a-80c4-9a7d4462d2f9" containerID="ac1ef05fa603dd69fa4d8d1d16e234f61492af355d24860a2e6e264a6cb2ed44" exitCode=0
Nov 03 08:03:12.333845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.333350    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn" event={"ID":"4fc257d6-34f4-412a-80c4-9a7d4462d2f9","Type":"ContainerDied","Data":"ac1ef05fa603dd69fa4d8d1d16e234f61492af355d24860a2e6e264a6cb2ed44"}
Nov 03 08:03:12.333901 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.333868    5070 scope.go:122] "RemoveContainer" containerID="ac1ef05fa603dd69fa4d8d1d16e234f61492af355d24860a2e6e264a6cb2ed44"
Nov 03 08:03:12.335504 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.335463    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:03:12.335627 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.335588    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:03:12.335681 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.335635    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:03:12.336718 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.336693    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt" event={"ID":"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc","Type":"ContainerStarted","Data":"9c17e76f83d02982413d20c13e208ed6683938d3c14d74b497e6d9aa053b41ff"}
Nov 03 08:03:12.685670 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.685634    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"]
Nov 03 08:03:12.720274 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.718512    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:12.720530 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.720501    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"]
Nov 03 08:03:12.734410 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.733843    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ffe7a8\"/\"default-dockercfg-vkqgz\""
Nov 03 08:03:12.738253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.737431    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ffe7a8\"/\"simulator-w5-ffe7a8-kserve-self-signed-certs\""
Nov 03 08:03:12.738253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.737543    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ffe7a8\"/\"openshift-service-ca.crt\""
Nov 03 08:03:12.738253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.737632    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ffe7a8\"/\"kube-root-ca.crt\""
Nov 03 08:03:12.789839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.789195    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e512c2-d43c-452c-86a3-7cf6bfea1558-tls-certs\") pod \"simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") " pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:12.789839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.789251    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-home\") pod \"simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") " pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:12.789839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.789302    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-tmp-dir\") pod \"simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") " pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:12.789839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.789416    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-dshm\") pod \"simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") " pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:12.789839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.789460    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-model-cache\") pod \"simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") " pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:12.789839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.789487    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lssjk\" (UniqueName: \"kubernetes.io/projected/77e512c2-d43c-452c-86a3-7cf6bfea1558-kube-api-access-lssjk\") pod \"simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") " pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:12.870202 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.870169    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:03:12.870568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.870545    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 03 08:03:12.870722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.870697    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt" podUID="f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 03 08:03:12.894086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.893456    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-tmp-dir\") pod \"simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") " pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:12.894086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.893559    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-dshm\") pod \"simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") " pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:12.894086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.893587    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-model-cache\") pod \"simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") " pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:12.894086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.893605    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lssjk\" (UniqueName: \"kubernetes.io/projected/77e512c2-d43c-452c-86a3-7cf6bfea1558-kube-api-access-lssjk\") pod \"simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") " pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:12.894086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.893661    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e512c2-d43c-452c-86a3-7cf6bfea1558-tls-certs\") pod \"simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") " pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:12.894086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.893689    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-home\") pod \"simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") " pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:12.894553 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.894145    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-home\") pod \"simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") " pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:12.894680 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.894656    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-tmp-dir\") pod \"simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") " pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:12.894920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.894898    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-model-cache\") pod \"simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") " pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:12.896869 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.896845    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-dshm\") pod \"simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") " pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:12.897917 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.897894    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e512c2-d43c-452c-86a3-7cf6bfea1558-tls-certs\") pod \"simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") " pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:12.900682 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:12.900660    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lssjk\" (UniqueName: \"kubernetes.io/projected/77e512c2-d43c-452c-86a3-7cf6bfea1558-kube-api-access-lssjk\") pod \"simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") " pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:13.043181 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:13.043091    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:13.346366 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:13.346272    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn" event={"ID":"4fc257d6-34f4-412a-80c4-9a7d4462d2f9","Type":"ContainerStarted","Data":"2f031fbd4e1e1df8facbdd8a5355bb69211f9a7fa837238cbdf0f9b26cee72aa"}
Nov 03 08:03:13.419782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:13.419722    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"]
Nov 03 08:03:13.420237 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:03:13.420198    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77e512c2_d43c_452c_86a3_7cf6bfea1558.slice/crio-bc8fa99e6de6d8edcef8f9a8e2fb14e2b54d67f125a8655fd6cffa3e65ab04a8 WatchSource:0}: Error finding container bc8fa99e6de6d8edcef8f9a8e2fb14e2b54d67f125a8655fd6cffa3e65ab04a8: Status 404 returned error can't find the container with id bc8fa99e6de6d8edcef8f9a8e2fb14e2b54d67f125a8655fd6cffa3e65ab04a8
Nov 03 08:03:13.514682 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:13.514592    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:03:13.516055 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:13.516025    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 08:03:13.516163 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:13.516085    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn" podUID="4fc257d6-34f4-412a-80c4-9a7d4462d2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 08:03:13.910817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:13.899548    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 03 08:03:13.910817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:13.899608    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt" podUID="f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 03 08:03:14.355060 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:14.355016    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" event={"ID":"77e512c2-d43c-452c-86a3-7cf6bfea1558","Type":"ContainerStarted","Data":"0d350a5683831cadc84463292d1d7791cbc2733f865752dfb67ebf35ffba5735"}
Nov 03 08:03:14.355664 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:14.355628    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" event={"ID":"77e512c2-d43c-452c-86a3-7cf6bfea1558","Type":"ContainerStarted","Data":"bc8fa99e6de6d8edcef8f9a8e2fb14e2b54d67f125a8655fd6cffa3e65ab04a8"}
Nov 03 08:03:14.398783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:14.395224    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podStartSLOduration=2.3952047260000002 podStartE2EDuration="2.395204726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:03:14.382628085 +0000 UTC m=+592.497092018" watchObservedRunningTime="2026-11-03 08:03:14.395204726 +0000 UTC m=+592.509668658"
Nov 03 08:03:14.525260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:14.518250    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 08:03:14.525260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:14.518346    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn" podUID="4fc257d6-34f4-412a-80c4-9a7d4462d2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 08:03:14.874784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:14.869156    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 03 08:03:14.874784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:14.869220    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt" podUID="f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 03 08:03:15.044459 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:15.044431    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:15.516691 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:15.515151    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 08:03:15.516691 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:15.515216    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn" podUID="4fc257d6-34f4-412a-80c4-9a7d4462d2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 08:03:15.875782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:15.875163    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 03 08:03:15.875782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:15.875229    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt" podUID="f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 03 08:03:16.515214 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:16.515178    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 08:03:16.515507 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:16.515474    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn" podUID="4fc257d6-34f4-412a-80c4-9a7d4462d2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 08:03:16.880783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:16.879215    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 03 08:03:16.880783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:16.879302    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt" podUID="f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 03 08:03:17.053774 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:17.053727    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:03:17.515735 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:17.515359    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 08:03:17.515735 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:17.515407    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn" podUID="4fc257d6-34f4-412a-80c4-9a7d4462d2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 08:03:17.773844 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:17.770230    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"]
Nov 03 08:03:17.818782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:17.818631    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"]
Nov 03 08:03:17.818782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:17.818738    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:17.822061 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:17.822038    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-19bb0a\"/\"kube-root-ca.crt\""
Nov 03 08:03:17.822431 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:17.822349    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-19bb0a\"/\"simulator-w1-19bb0a-kserve-self-signed-certs\""
Nov 03 08:03:17.822596 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:17.822576    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-19bb0a\"/\"openshift-service-ca.crt\""
Nov 03 08:03:17.822721 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:17.822704    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-19bb0a\"/\"default-dockercfg-s8gbj\""
Nov 03 08:03:17.880785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:17.869416    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Nov 03 08:03:17.880785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:17.869471    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt" podUID="f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Nov 03 08:03:17.922434 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:17.922401    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-tmp-dir\") pod \"simulator-w1-19bb0a-kserve-84b8688499-5nfnl\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") " pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:17.922628 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:17.922461    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-home\") pod \"simulator-w1-19bb0a-kserve-84b8688499-5nfnl\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") " pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:17.922628 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:17.922515    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-model-cache\") pod \"simulator-w1-19bb0a-kserve-84b8688499-5nfnl\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") " pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:17.922628 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:17.922570    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09a6416b-7395-4d6e-a6e5-45cf59873c6b-tls-certs\") pod \"simulator-w1-19bb0a-kserve-84b8688499-5nfnl\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") " pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:17.922628 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:17.922608    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-464tx\" (UniqueName: \"kubernetes.io/projected/09a6416b-7395-4d6e-a6e5-45cf59873c6b-kube-api-access-464tx\") pod \"simulator-w1-19bb0a-kserve-84b8688499-5nfnl\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") " pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:17.922863 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:17.922738    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-dshm\") pod \"simulator-w1-19bb0a-kserve-84b8688499-5nfnl\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") " pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:18.023450 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.023411    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-tmp-dir\") pod \"simulator-w1-19bb0a-kserve-84b8688499-5nfnl\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") " pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:18.023635 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.023462    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-home\") pod \"simulator-w1-19bb0a-kserve-84b8688499-5nfnl\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") " pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:18.023635 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.023480    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-model-cache\") pod \"simulator-w1-19bb0a-kserve-84b8688499-5nfnl\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") " pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:18.023635 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.023528    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09a6416b-7395-4d6e-a6e5-45cf59873c6b-tls-certs\") pod \"simulator-w1-19bb0a-kserve-84b8688499-5nfnl\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") " pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:18.023635 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.023562    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-464tx\" (UniqueName: \"kubernetes.io/projected/09a6416b-7395-4d6e-a6e5-45cf59873c6b-kube-api-access-464tx\") pod \"simulator-w1-19bb0a-kserve-84b8688499-5nfnl\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") " pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:18.023635 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.023607    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-dshm\") pod \"simulator-w1-19bb0a-kserve-84b8688499-5nfnl\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") " pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:18.023981 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.023883    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-tmp-dir\") pod \"simulator-w1-19bb0a-kserve-84b8688499-5nfnl\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") " pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:18.023981 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.023929    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-home\") pod \"simulator-w1-19bb0a-kserve-84b8688499-5nfnl\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") " pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:18.024320 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.024287    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-model-cache\") pod \"simulator-w1-19bb0a-kserve-84b8688499-5nfnl\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") " pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:18.025894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.025874    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-dshm\") pod \"simulator-w1-19bb0a-kserve-84b8688499-5nfnl\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") " pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:18.026310 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.026287    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09a6416b-7395-4d6e-a6e5-45cf59873c6b-tls-certs\") pod \"simulator-w1-19bb0a-kserve-84b8688499-5nfnl\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") " pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:18.033374 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.033355    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-464tx\" (UniqueName: \"kubernetes.io/projected/09a6416b-7395-4d6e-a6e5-45cf59873c6b-kube-api-access-464tx\") pod \"simulator-w1-19bb0a-kserve-84b8688499-5nfnl\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") " pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:18.074302 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.073832    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:03:18.074532 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.074508    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 08:03:18.074678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.074656    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p" podUID="4f3b3503-699a-4697-b121-351ad81bad8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 08:03:18.132330 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.132298    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:18.427595 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.427563    5070 generic.go:356] "Generic (PLEG): container finished" podID="4f3b3503-699a-4697-b121-351ad81bad8a" containerID="776e6bbde0a6b87bf9d2c28e2d8a298aeabc01552cdab214a0425efce769f8f3" exitCode=0
Nov 03 08:03:18.427803 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.427634    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p" event={"ID":"4f3b3503-699a-4697-b121-351ad81bad8a","Type":"ContainerDied","Data":"776e6bbde0a6b87bf9d2c28e2d8a298aeabc01552cdab214a0425efce769f8f3"}
Nov 03 08:03:18.428108 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.428091    5070 scope.go:122] "RemoveContainer" containerID="776e6bbde0a6b87bf9d2c28e2d8a298aeabc01552cdab214a0425efce769f8f3"
Nov 03 08:03:18.429354 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.429327    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:03:18.429421 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.429395    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:03:18.429455 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.429426    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:03:18.437580 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.437558    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"]
Nov 03 08:03:18.439490 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:03:18.439469    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09a6416b_7395_4d6e_a6e5_45cf59873c6b.slice/crio-1e0b634b2da9c5155714a52ae7e6703445b8a86470fe5444cc24540f392ec887 WatchSource:0}: Error finding container 1e0b634b2da9c5155714a52ae7e6703445b8a86470fe5444cc24540f392ec887: Status 404 returned error can't find the container with id 1e0b634b2da9c5155714a52ae7e6703445b8a86470fe5444cc24540f392ec887
Nov 03 08:03:18.515283 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.515254    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Nov 03 08:03:18.515403 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.515313    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn" podUID="4fc257d6-34f4-412a-80c4-9a7d4462d2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Nov 03 08:03:18.871807 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.871769    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:03:18.872861 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:18.872839    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:03:19.436289 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:19.436249    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p" event={"ID":"4f3b3503-699a-4697-b121-351ad81bad8a","Type":"ContainerStarted","Data":"41f9ed86772f153450c891f4cf3bbbd6ca5a5c633806e8d922f1621e3a16c7ef"}
Nov 03 08:03:19.438127 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:19.438099    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" event={"ID":"09a6416b-7395-4d6e-a6e5-45cf59873c6b","Type":"ContainerStarted","Data":"9c8106957f21cc7aa15ec81b8ce04508b85a41a78dabc8ff5e87c36cafc90bd0"}
Nov 03 08:03:19.438273 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:19.438138    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" event={"ID":"09a6416b-7395-4d6e-a6e5-45cf59873c6b","Type":"ContainerStarted","Data":"1e0b634b2da9c5155714a52ae7e6703445b8a86470fe5444cc24540f392ec887"}
Nov 03 08:03:19.500555 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:19.500492    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podStartSLOduration=2.500473381 podStartE2EDuration="2.500473381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:03:19.497569679 +0000 UTC m=+597.612033610" watchObservedRunningTime="2026-11-03 08:03:19.500473381 +0000 UTC m=+597.614937314"
Nov 03 08:03:19.539610 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:19.539575    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:03:19.545496 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:19.545464    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:03:19.764835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:19.764141    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:03:19.764835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:19.764319    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 08:03:19.764835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:19.764376    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p" podUID="4f3b3503-699a-4697-b121-351ad81bad8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 08:03:20.132589 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:20.132561    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:20.781226 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:20.781163    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 08:03:20.781727 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:20.781253    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p" podUID="4f3b3503-699a-4697-b121-351ad81bad8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 08:03:21.767782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:21.761072    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 08:03:21.767782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:21.761134    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p" podUID="4f3b3503-699a-4697-b121-351ad81bad8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 08:03:22.101451 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:03:22.101414    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:03:22.130051 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:22.130021    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/2.log"
Nov 03 08:03:22.130806 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:22.130783    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/2.log"
Nov 03 08:03:22.761096 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:22.761060    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 08:03:22.761357 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:22.761329    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p" podUID="4f3b3503-699a-4697-b121-351ad81bad8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 08:03:23.043575 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:23.043462    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:23.071058 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:23.071020    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:23.512189 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:23.512159    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:03:23.780962 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:23.780870    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Nov 03 08:03:23.780962 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:23.780932    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p" podUID="4f3b3503-699a-4697-b121-351ad81bad8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Nov 03 08:03:23.987044 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:23.986994    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"]
Nov 03 08:03:23.991580 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:23.991550    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"]
Nov 03 08:03:23.991741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:23.991660    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:23.995634 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:23.995604    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-15a633\"/\"default-dockercfg-mgwxq\""
Nov 03 08:03:23.995896 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:23.995878    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-15a633\"/\"simulator-w4-15a633-kserve-self-signed-certs\""
Nov 03 08:03:23.996332 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:23.996091    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-15a633\"/\"kube-root-ca.crt\""
Nov 03 08:03:23.996519 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:23.996502    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-15a633\"/\"openshift-service-ca.crt\""
Nov 03 08:03:24.112916 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.112879    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c00158-d927-478c-8fee-13f94bbdb2d6-tls-certs\") pod \"simulator-w4-15a633-kserve-55b977b85d-lgpmz\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") " pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:24.113085 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.112929    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-dshm\") pod \"simulator-w4-15a633-kserve-55b977b85d-lgpmz\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") " pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:24.113085 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.113055    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-home\") pod \"simulator-w4-15a633-kserve-55b977b85d-lgpmz\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") " pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:24.113219 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.113196    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-model-cache\") pod \"simulator-w4-15a633-kserve-55b977b85d-lgpmz\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") " pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:24.113339 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.113319    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rz6w\" (UniqueName: \"kubernetes.io/projected/54c00158-d927-478c-8fee-13f94bbdb2d6-kube-api-access-5rz6w\") pod \"simulator-w4-15a633-kserve-55b977b85d-lgpmz\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") " pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:24.113376 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.113361    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-tmp-dir\") pod \"simulator-w4-15a633-kserve-55b977b85d-lgpmz\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") " pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:24.214372 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.214335    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-model-cache\") pod \"simulator-w4-15a633-kserve-55b977b85d-lgpmz\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") " pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:24.214534 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.214433    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rz6w\" (UniqueName: \"kubernetes.io/projected/54c00158-d927-478c-8fee-13f94bbdb2d6-kube-api-access-5rz6w\") pod \"simulator-w4-15a633-kserve-55b977b85d-lgpmz\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") " pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:24.214534 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.214459    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-tmp-dir\") pod \"simulator-w4-15a633-kserve-55b977b85d-lgpmz\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") " pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:24.214534 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.214491    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c00158-d927-478c-8fee-13f94bbdb2d6-tls-certs\") pod \"simulator-w4-15a633-kserve-55b977b85d-lgpmz\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") " pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:24.214534 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.214516    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-dshm\") pod \"simulator-w4-15a633-kserve-55b977b85d-lgpmz\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") " pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:24.214679 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.214560    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-home\") pod \"simulator-w4-15a633-kserve-55b977b85d-lgpmz\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") " pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:24.215071 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.215045    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-tmp-dir\") pod \"simulator-w4-15a633-kserve-55b977b85d-lgpmz\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") " pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:24.215209 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.215128    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-home\") pod \"simulator-w4-15a633-kserve-55b977b85d-lgpmz\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") " pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:24.215209 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.215154    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-model-cache\") pod \"simulator-w4-15a633-kserve-55b977b85d-lgpmz\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") " pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:24.217150 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.217129    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-dshm\") pod \"simulator-w4-15a633-kserve-55b977b85d-lgpmz\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") " pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:24.217378 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.217357    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c00158-d927-478c-8fee-13f94bbdb2d6-tls-certs\") pod \"simulator-w4-15a633-kserve-55b977b85d-lgpmz\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") " pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:24.221728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.221704    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rz6w\" (UniqueName: \"kubernetes.io/projected/54c00158-d927-478c-8fee-13f94bbdb2d6-kube-api-access-5rz6w\") pod \"simulator-w4-15a633-kserve-55b977b85d-lgpmz\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") " pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:24.307182 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.307152    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:24.594750 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.594717    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"]
Nov 03 08:03:24.596112 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:03:24.596084    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54c00158_d927_478c_8fee_13f94bbdb2d6.slice/crio-dcbd6ff05692bf9148a384617563bfbbe87d9d7f0a1bb4aa22d44931b539f973 WatchSource:0}: Error finding container dcbd6ff05692bf9148a384617563bfbbe87d9d7f0a1bb4aa22d44931b539f973: Status 404 returned error can't find the container with id dcbd6ff05692bf9148a384617563bfbbe87d9d7f0a1bb4aa22d44931b539f973
Nov 03 08:03:24.764848 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.764819    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:03:24.765113 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.765090    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:03:24.765734 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:24.765718    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:03:25.169052 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:25.169022    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:03:25.519277 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:25.516696    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" event={"ID":"54c00158-d927-478c-8fee-13f94bbdb2d6","Type":"ContainerStarted","Data":"3858e4e8bb39a20b059d2b818825713fc649efe9b8710271688eca56b6ae2aa3"}
Nov 03 08:03:25.519277 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:25.516751    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" event={"ID":"54c00158-d927-478c-8fee-13f94bbdb2d6","Type":"ContainerStarted","Data":"dcbd6ff05692bf9148a384617563bfbbe87d9d7f0a1bb4aa22d44931b539f973"}
Nov 03 08:03:25.532921 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:25.532871    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podStartSLOduration=2.532857278 podStartE2EDuration="2.532857278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:03:25.530671876 +0000 UTC m=+603.645135807" watchObservedRunningTime="2026-11-03 08:03:25.532857278 +0000 UTC m=+603.647321209"
Nov 03 08:03:26.307493 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:26.307457    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:28.133375 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:28.133343    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:28.147861 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:28.147836    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:28.545185 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:28.545111    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:03:34.307663 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:34.307623    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:34.320095 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:34.320071    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:34.581729 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:34.581660    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:03:35.160253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.160217    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"]
Nov 03 08:03:35.165652 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.165630    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:35.170821 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.170796    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ffe7a8\"/\"premium-w5-ffe7a8-kserve-self-signed-certs\""
Nov 03 08:03:35.175325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.175301    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"]
Nov 03 08:03:35.234004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.233965    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-home\") pod \"premium-w5-ffe7a8-kserve-79db785d58-2skmc\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") " pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:35.234004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.234005    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmmdp\" (UniqueName: \"kubernetes.io/projected/381033e1-f38c-48ba-9bf2-45ba216df6fb-kube-api-access-qmmdp\") pod \"premium-w5-ffe7a8-kserve-79db785d58-2skmc\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") " pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:35.234254 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.234099    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-tmp-dir\") pod \"premium-w5-ffe7a8-kserve-79db785d58-2skmc\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") " pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:35.234325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.234306    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/381033e1-f38c-48ba-9bf2-45ba216df6fb-tls-certs\") pod \"premium-w5-ffe7a8-kserve-79db785d58-2skmc\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") " pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:35.234387 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.234350    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-model-cache\") pod \"premium-w5-ffe7a8-kserve-79db785d58-2skmc\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") " pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:35.234451 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.234430    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-dshm\") pod \"premium-w5-ffe7a8-kserve-79db785d58-2skmc\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") " pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:35.335573 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.335539    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/381033e1-f38c-48ba-9bf2-45ba216df6fb-tls-certs\") pod \"premium-w5-ffe7a8-kserve-79db785d58-2skmc\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") " pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:35.335573 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.335589    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-model-cache\") pod \"premium-w5-ffe7a8-kserve-79db785d58-2skmc\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") " pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:35.335573 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.335628    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-dshm\") pod \"premium-w5-ffe7a8-kserve-79db785d58-2skmc\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") " pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:35.336427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.335696    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-home\") pod \"premium-w5-ffe7a8-kserve-79db785d58-2skmc\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") " pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:35.336427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.335722    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmmdp\" (UniqueName: \"kubernetes.io/projected/381033e1-f38c-48ba-9bf2-45ba216df6fb-kube-api-access-qmmdp\") pod \"premium-w5-ffe7a8-kserve-79db785d58-2skmc\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") " pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:35.336427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.335783    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-tmp-dir\") pod \"premium-w5-ffe7a8-kserve-79db785d58-2skmc\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") " pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:35.336427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.336206    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-home\") pod \"premium-w5-ffe7a8-kserve-79db785d58-2skmc\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") " pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:35.336427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.336318    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-tmp-dir\") pod \"premium-w5-ffe7a8-kserve-79db785d58-2skmc\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") " pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:35.336427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.336350    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-model-cache\") pod \"premium-w5-ffe7a8-kserve-79db785d58-2skmc\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") " pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:35.338091 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.338068    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-dshm\") pod \"premium-w5-ffe7a8-kserve-79db785d58-2skmc\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") " pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:35.339000 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.338980    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/381033e1-f38c-48ba-9bf2-45ba216df6fb-tls-certs\") pod \"premium-w5-ffe7a8-kserve-79db785d58-2skmc\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") " pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:35.344953 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.344934    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmmdp\" (UniqueName: \"kubernetes.io/projected/381033e1-f38c-48ba-9bf2-45ba216df6fb-kube-api-access-qmmdp\") pod \"premium-w5-ffe7a8-kserve-79db785d58-2skmc\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") " pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:35.483730 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.483648    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:35.767556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:35.767521    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"]
Nov 03 08:03:35.768370 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:03:35.768346    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod381033e1_f38c_48ba_9bf2_45ba216df6fb.slice/crio-8be46b1b9934259d609f54d2b2f42a7af5af1cd4f77e7af1c9d31470ecc9a2a8 WatchSource:0}: Error finding container 8be46b1b9934259d609f54d2b2f42a7af5af1cd4f77e7af1c9d31470ecc9a2a8: Status 404 returned error can't find the container with id 8be46b1b9934259d609f54d2b2f42a7af5af1cd4f77e7af1c9d31470ecc9a2a8
Nov 03 08:03:36.585933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:36.585895    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" event={"ID":"381033e1-f38c-48ba-9bf2-45ba216df6fb","Type":"ContainerStarted","Data":"903c85d9d89bb11fe84693e4fbf2ac4e38b88fc9b1fbbacd35df488b83412824"}
Nov 03 08:03:36.586323 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:36.585944    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" event={"ID":"381033e1-f38c-48ba-9bf2-45ba216df6fb","Type":"ContainerStarted","Data":"8be46b1b9934259d609f54d2b2f42a7af5af1cd4f77e7af1c9d31470ecc9a2a8"}
Nov 03 08:03:36.602339 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:36.602286    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podStartSLOduration=1.6022724080000001 podStartE2EDuration="1.602272408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:03:36.600811599 +0000 UTC m=+614.715275527" watchObservedRunningTime="2026-11-03 08:03:36.602272408 +0000 UTC m=+614.716736342"
Nov 03 08:03:37.484493 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:37.484461    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:40.785069 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:40.784107    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:03:45.483872 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:45.483843    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:45.496575 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:45.496550    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:45.657945 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:45.657915    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:03:52.111778 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:03:52.099990    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:03:55.462532 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.462496    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"]
Nov 03 08:03:55.468052 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.468031    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:03:55.474663 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.474628    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"]
Nov 03 08:03:55.475065 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.474946    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-15a633\"/\"premium-w4-15a633-kserve-self-signed-certs\""
Nov 03 08:03:55.485571 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.485350    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fed520-3264-4574-bafa-ac12dde22a02-tls-certs\") pod \"premium-w4-15a633-kserve-75575b7cf5-qzmk7\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") " pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:03:55.485571 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.485434    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-tmp-dir\") pod \"premium-w4-15a633-kserve-75575b7cf5-qzmk7\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") " pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:03:55.485712 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.485625    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-model-cache\") pod \"premium-w4-15a633-kserve-75575b7cf5-qzmk7\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") " pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:03:55.485712 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.485658    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-home\") pod \"premium-w4-15a633-kserve-75575b7cf5-qzmk7\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") " pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:03:55.485712 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.485687    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-dshm\") pod \"premium-w4-15a633-kserve-75575b7cf5-qzmk7\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") " pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:03:55.486082 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.485714    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gzqq\" (UniqueName: \"kubernetes.io/projected/a5fed520-3264-4574-bafa-ac12dde22a02-kube-api-access-8gzqq\") pod \"premium-w4-15a633-kserve-75575b7cf5-qzmk7\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") " pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:03:55.587585 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.587546    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-model-cache\") pod \"premium-w4-15a633-kserve-75575b7cf5-qzmk7\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") " pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:03:55.587825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.587596    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-home\") pod \"premium-w4-15a633-kserve-75575b7cf5-qzmk7\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") " pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:03:55.587825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.587652    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-dshm\") pod \"premium-w4-15a633-kserve-75575b7cf5-qzmk7\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") " pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:03:55.587825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.587683    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gzqq\" (UniqueName: \"kubernetes.io/projected/a5fed520-3264-4574-bafa-ac12dde22a02-kube-api-access-8gzqq\") pod \"premium-w4-15a633-kserve-75575b7cf5-qzmk7\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") " pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:03:55.587825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.587789    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fed520-3264-4574-bafa-ac12dde22a02-tls-certs\") pod \"premium-w4-15a633-kserve-75575b7cf5-qzmk7\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") " pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:03:55.588083 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.587826    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-tmp-dir\") pod \"premium-w4-15a633-kserve-75575b7cf5-qzmk7\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") " pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:03:55.588360 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.588336    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-tmp-dir\") pod \"premium-w4-15a633-kserve-75575b7cf5-qzmk7\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") " pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:03:55.588590 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.588571    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-model-cache\") pod \"premium-w4-15a633-kserve-75575b7cf5-qzmk7\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") " pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:03:55.588861 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.588808    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-home\") pod \"premium-w4-15a633-kserve-75575b7cf5-qzmk7\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") " pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:03:55.595239 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.592315    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-dshm\") pod \"premium-w4-15a633-kserve-75575b7cf5-qzmk7\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") " pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:03:55.596452 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.596076    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fed520-3264-4574-bafa-ac12dde22a02-tls-certs\") pod \"premium-w4-15a633-kserve-75575b7cf5-qzmk7\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") " pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:03:55.598491 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.598466    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gzqq\" (UniqueName: \"kubernetes.io/projected/a5fed520-3264-4574-bafa-ac12dde22a02-kube-api-access-8gzqq\") pod \"premium-w4-15a633-kserve-75575b7cf5-qzmk7\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") " pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:03:55.782407 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:55.782333    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:03:56.054239 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:56.054208    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"]
Nov 03 08:03:56.055649 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:03:56.055620    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5fed520_3264_4574_bafa_ac12dde22a02.slice/crio-01132aebca33318c8af6200e6fcf153c77dd48852bfb0ae10a08ae6f997def21 WatchSource:0}: Error finding container 01132aebca33318c8af6200e6fcf153c77dd48852bfb0ae10a08ae6f997def21: Status 404 returned error can't find the container with id 01132aebca33318c8af6200e6fcf153c77dd48852bfb0ae10a08ae6f997def21
Nov 03 08:03:56.716419 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:56.716386    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" event={"ID":"a5fed520-3264-4574-bafa-ac12dde22a02","Type":"ContainerStarted","Data":"d387b38ddd3da9e703275063b08459ba2a68b950ab10f4b9d53dd75cc4752003"}
Nov 03 08:03:56.716419 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:56.716426    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" event={"ID":"a5fed520-3264-4574-bafa-ac12dde22a02","Type":"ContainerStarted","Data":"01132aebca33318c8af6200e6fcf153c77dd48852bfb0ae10a08ae6f997def21"}
Nov 03 08:03:56.730610 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:56.730571    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podStartSLOduration=1.730560633 podStartE2EDuration="1.730560633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:03:56.729975607 +0000 UTC m=+634.844439541" watchObservedRunningTime="2026-11-03 08:03:56.730560633 +0000 UTC m=+634.845024557"
Nov 03 08:03:56.783005 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:56.782989    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:03:58.418296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:03:58.418266    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:04:00.395561 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.395528    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"]
Nov 03 08:04:00.401067 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.401044    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.407333 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.407305    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-3e88b7-openshift-default-dockercfg-k6k9p\""
Nov 03 08:04:00.411274 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.411249    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"]
Nov 03 08:04:00.442485 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.442458    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.442625 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.442493    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.442625 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.442522    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bnsb\" (UniqueName: \"kubernetes.io/projected/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-kube-api-access-4bnsb\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.442625 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.442608    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.442813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.442700    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.442813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.442737    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.442932 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.442856    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.442932 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.442901    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.443036 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.442936    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.543905 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.543876    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.544032 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.543913    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.544032 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.543959    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bnsb\" (UniqueName: \"kubernetes.io/projected/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-kube-api-access-4bnsb\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.544032 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.543985    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.544191 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.544066    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.544191 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.544110    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.544191 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.544161    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.544352 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.544196    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.544352 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.544229    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.544541 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.544510    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.544616 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.544544    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.544616 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.544600    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.544886 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.544867    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.545279 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.545210    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.546374 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.546354    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.546801 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.546782    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.550861 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.550841    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.550971 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.550879    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bnsb\" (UniqueName: \"kubernetes.io/projected/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-kube-api-access-4bnsb\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.716302 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.716226    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:00.995516 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.995488    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"]
Nov 03 08:04:00.996259 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:04:00.996231    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbb0b5e6_9abd_45f5_b444_572bd1a53fc1.slice/crio-21e91b281c1fba8ad94876fe3dcd1ac0078ca33e256cf151d5a437faef0778a9 WatchSource:0}: Error finding container 21e91b281c1fba8ad94876fe3dcd1ac0078ca33e256cf151d5a437faef0778a9: Status 404 returned error can't find the container with id 21e91b281c1fba8ad94876fe3dcd1ac0078ca33e256cf151d5a437faef0778a9
Nov 03 08:04:00.998718 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.998683    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:04:00.998804 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.998776    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:04:00.998856 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:00.998810    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:04:01.750503 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:01.750462    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89" event={"ID":"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1","Type":"ContainerStarted","Data":"b1401b6134a3ead2bdae6e4103b8040456f68f7acc759b0a14112e64ab659688"}
Nov 03 08:04:01.750922 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:01.750529    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89" event={"ID":"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1","Type":"ContainerStarted","Data":"21e91b281c1fba8ad94876fe3dcd1ac0078ca33e256cf151d5a437faef0778a9"}
Nov 03 08:04:01.774111 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:01.774063    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89" podStartSLOduration=1.774051625 podStartE2EDuration="1.774051625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:04:01.771669469 +0000 UTC m=+639.886133400" watchObservedRunningTime="2026-11-03 08:04:01.774051625 +0000 UTC m=+639.888515556"
Nov 03 08:04:02.716826 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:02.716753    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:02.718231 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:02.718195    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 08:04:02.718393 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:02.718248    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89" podUID="bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 08:04:03.717077 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:03.717045    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 08:04:03.717548 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:03.717095    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89" podUID="bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 08:04:04.718630 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:04.718586    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 08:04:04.719152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:04.718674    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89" podUID="bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 08:04:05.716947 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:05.716902    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 08:04:05.717143 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:05.716954    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89" podUID="bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 08:04:05.783395 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:05.783368    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:04:05.797269 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:05.797243    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:04:05.808407 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:05.808386    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:04:06.717100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:06.717069    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 08:04:06.717266 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:06.717125    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89" podUID="bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 08:04:07.716559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:07.716525    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Nov 03 08:04:07.716988 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:07.716574    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89" podUID="bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Nov 03 08:04:08.722685 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:08.722655    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:08.723104 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:08.722999    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:08.723779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:08.723738    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:04:14.910198 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:14.910161    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h"]
Nov 03 08:04:14.914446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:14.914423    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h"
Nov 03 08:04:14.921600 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:14.921578    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:04:14.921909 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:14.921865    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:04:14.922038 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:14.921939    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:04:14.922038 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:14.921943    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:04:14.923879 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:14.923857    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr5gx\" (UniqueName: \"kubernetes.io/projected/9966eba6-1242-4af5-9fc3-3a906ecac70e-kube-api-access-vr5gx\") pod \"maas-api-revoke-sub-c4df7568-h4l6h\" (UID: \"9966eba6-1242-4af5-9fc3-3a906ecac70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h"
Nov 03 08:04:14.924143 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:14.924119    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9966eba6-1242-4af5-9fc3-3a906ecac70e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4df7568-h4l6h\" (UID: \"9966eba6-1242-4af5-9fc3-3a906ecac70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h"
Nov 03 08:04:14.924246 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:14.924222    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9966eba6-1242-4af5-9fc3-3a906ecac70e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4df7568-h4l6h\" (UID: \"9966eba6-1242-4af5-9fc3-3a906ecac70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h"
Nov 03 08:04:14.926161 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:14.926137    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h"]
Nov 03 08:04:15.025844 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.025811    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr5gx\" (UniqueName: \"kubernetes.io/projected/9966eba6-1242-4af5-9fc3-3a906ecac70e-kube-api-access-vr5gx\") pod \"maas-api-revoke-sub-c4df7568-h4l6h\" (UID: \"9966eba6-1242-4af5-9fc3-3a906ecac70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h"
Nov 03 08:04:15.026013 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.025882    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9966eba6-1242-4af5-9fc3-3a906ecac70e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4df7568-h4l6h\" (UID: \"9966eba6-1242-4af5-9fc3-3a906ecac70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h"
Nov 03 08:04:15.026013 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.025953    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9966eba6-1242-4af5-9fc3-3a906ecac70e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4df7568-h4l6h\" (UID: \"9966eba6-1242-4af5-9fc3-3a906ecac70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h"
Nov 03 08:04:15.026833 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.026802    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9966eba6-1242-4af5-9fc3-3a906ecac70e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4df7568-h4l6h\" (UID: \"9966eba6-1242-4af5-9fc3-3a906ecac70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h"
Nov 03 08:04:15.028302 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.028271    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9966eba6-1242-4af5-9fc3-3a906ecac70e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4df7568-h4l6h\" (UID: \"9966eba6-1242-4af5-9fc3-3a906ecac70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h"
Nov 03 08:04:15.033404 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.033377    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr5gx\" (UniqueName: \"kubernetes.io/projected/9966eba6-1242-4af5-9fc3-3a906ecac70e-kube-api-access-vr5gx\") pod \"maas-api-revoke-sub-c4df7568-h4l6h\" (UID: \"9966eba6-1242-4af5-9fc3-3a906ecac70e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h"
Nov 03 08:04:15.235204 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.235122    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h"
Nov 03 08:04:15.262783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.258909    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"]
Nov 03 08:04:15.266030 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.265988    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.291963 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.291928    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-76ef13-openshift-default-dockercfg-4ftn9\""
Nov 03 08:04:15.292630 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.292607    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"]
Nov 03 08:04:15.328692 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.328659    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.328817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.328728    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.328893 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.328836    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.328893 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.328887    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.329009 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.328940    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.329009 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.328979    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.329121 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.329020    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdxn7\" (UniqueName: \"kubernetes.io/projected/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-kube-api-access-hdxn7\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.329121 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.329098    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.329274 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.329129    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.430886 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.430849    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.431071 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.430921    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.431071 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.431028    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.431071 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.431061    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.431257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.431116    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.431318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.431229    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.431384 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.431312    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdxn7\" (UniqueName: \"kubernetes.io/projected/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-kube-api-access-hdxn7\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.431438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.431402    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.431438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.431430    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.431605 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.431537    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.431672 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.431600    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.432068 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.431806    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.432068 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.432024    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.432239 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.432208    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.433933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.433904    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.434462 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.434442    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.437873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.437851    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.438086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.438066    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdxn7\" (UniqueName: \"kubernetes.io/projected/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-kube-api-access-hdxn7\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.553461 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.553383    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h"]
Nov 03 08:04:15.560257 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:04:15.560216    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9966eba6_1242_4af5_9fc3_3a906ecac70e.slice/crio-97ee61e98e42eb653f7c6e665d02268030e211228b4ae4094dd8211483a1f45e WatchSource:0}: Error finding container 97ee61e98e42eb653f7c6e665d02268030e211228b4ae4094dd8211483a1f45e: Status 404 returned error can't find the container with id 97ee61e98e42eb653f7c6e665d02268030e211228b4ae4094dd8211483a1f45e
Nov 03 08:04:15.562744 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.562725    5070 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 08:04:15.609287 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.609261    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:15.865422 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.865377    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h" event={"ID":"9966eba6-1242-4af5-9fc3-3a906ecac70e","Type":"ContainerStarted","Data":"97ee61e98e42eb653f7c6e665d02268030e211228b4ae4094dd8211483a1f45e"}
Nov 03 08:04:15.893321 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.893295    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"]
Nov 03 08:04:15.893920 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:04:15.893888    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7faa6ae_27c1_4a98_8c83_4a3fb1669835.slice/crio-9d0091b34ea1d8cc7285fee5f2a01d7039b24bc0fb8c8d6217df2b863bc63f1d WatchSource:0}: Error finding container 9d0091b34ea1d8cc7285fee5f2a01d7039b24bc0fb8c8d6217df2b863bc63f1d: Status 404 returned error can't find the container with id 9d0091b34ea1d8cc7285fee5f2a01d7039b24bc0fb8c8d6217df2b863bc63f1d
Nov 03 08:04:15.896558 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.896514    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:04:15.896646 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.896607    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:04:15.896688 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:15.896653    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:04:16.902183 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:16.902141    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx" event={"ID":"d7faa6ae-27c1-4a98-8c83-4a3fb1669835","Type":"ContainerStarted","Data":"020923369facd8800e378661dd8b7b52e1428e0e7aa851b1044a81716cc139f8"}
Nov 03 08:04:16.902962 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:16.902931    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx" event={"ID":"d7faa6ae-27c1-4a98-8c83-4a3fb1669835","Type":"ContainerStarted","Data":"9d0091b34ea1d8cc7285fee5f2a01d7039b24bc0fb8c8d6217df2b863bc63f1d"}
Nov 03 08:04:17.610459 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:17.610425    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:17.612548 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:17.612522    5070 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 08:04:17.612681 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:17.612578    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx" podUID="d7faa6ae-27c1-4a98-8c83-4a3fb1669835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 08:04:18.610151 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:18.610116    5070 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 08:04:18.610651 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:18.610176    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx" podUID="d7faa6ae-27c1-4a98-8c83-4a3fb1669835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 08:04:19.609874 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:19.609842    5070 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 08:04:19.610635 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:19.610592    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx" podUID="d7faa6ae-27c1-4a98-8c83-4a3fb1669835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 08:04:19.923572 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:19.923532    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h" event={"ID":"9966eba6-1242-4af5-9fc3-3a906ecac70e","Type":"ContainerStarted","Data":"e8da2e9d914e725f2254749300c54fbcc682df4a346d5646465518b8b184879f"}
Nov 03 08:04:19.937872 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:19.937598    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h" podStartSLOduration=2.4898244099999998 podStartE2EDuration="5.937582726s" totalImagesPullingTime="3.447758316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:04:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:04:19.936491947 +0000 UTC m=+658.050955877" watchObservedRunningTime="2026-11-03 08:04:19.937582726 +0000 UTC m=+658.052046658"
Nov 03 08:04:19.938970 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:19.938935    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx" podStartSLOduration=4.938924591 podStartE2EDuration="4.938924591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:04:16.922897464 +0000 UTC m=+655.037361395" watchObservedRunningTime="2026-11-03 08:04:19.938924591 +0000 UTC m=+658.053388522"
Nov 03 08:04:20.610586 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:20.610551    5070 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 08:04:20.610797 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:20.610619    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx" podUID="d7faa6ae-27c1-4a98-8c83-4a3fb1669835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 08:04:20.931403 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:20.931318    5070 generic.go:356] "Generic (PLEG): container finished" podID="9966eba6-1242-4af5-9fc3-3a906ecac70e" containerID="e8da2e9d914e725f2254749300c54fbcc682df4a346d5646465518b8b184879f" exitCode=0
Nov 03 08:04:20.931547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:20.931419    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h" event={"ID":"9966eba6-1242-4af5-9fc3-3a906ecac70e","Type":"ContainerDied","Data":"e8da2e9d914e725f2254749300c54fbcc682df4a346d5646465518b8b184879f"}
Nov 03 08:04:21.610676 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:21.610643    5070 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Nov 03 08:04:21.610891 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:21.610697    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx" podUID="d7faa6ae-27c1-4a98-8c83-4a3fb1669835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Nov 03 08:04:22.071707 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:04:22.071646    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:04:22.113095 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:22.113069    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h"
Nov 03 08:04:22.152799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:22.152776    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9966eba6-1242-4af5-9fc3-3a906ecac70e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9966eba6-1242-4af5-9fc3-3a906ecac70e-maas-api-openshift-ca\") pod \"9966eba6-1242-4af5-9fc3-3a906ecac70e\" (UID: \"9966eba6-1242-4af5-9fc3-3a906ecac70e\") "
Nov 03 08:04:22.152925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:22.152891    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9966eba6-1242-4af5-9fc3-3a906ecac70e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9966eba6-1242-4af5-9fc3-3a906ecac70e-maas-api-xks-ca\") pod \"9966eba6-1242-4af5-9fc3-3a906ecac70e\" (UID: \"9966eba6-1242-4af5-9fc3-3a906ecac70e\") "
Nov 03 08:04:22.152975 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:22.152940    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9966eba6-1242-4af5-9fc3-3a906ecac70e-kube-api-access-vr5gx\" (UniqueName: \"kubernetes.io/projected/9966eba6-1242-4af5-9fc3-3a906ecac70e-kube-api-access-vr5gx\") pod \"9966eba6-1242-4af5-9fc3-3a906ecac70e\" (UID: \"9966eba6-1242-4af5-9fc3-3a906ecac70e\") "
Nov 03 08:04:22.153148 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:22.153123    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9966eba6-1242-4af5-9fc3-3a906ecac70e-maas-api-openshift-ca" pod "9966eba6-1242-4af5-9fc3-3a906ecac70e" (UID: "9966eba6-1242-4af5-9fc3-3a906ecac70e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:04:22.153421 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:22.153394    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9966eba6-1242-4af5-9fc3-3a906ecac70e-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:22.155716 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:22.155691    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9966eba6-1242-4af5-9fc3-3a906ecac70e-maas-api-xks-ca" pod "9966eba6-1242-4af5-9fc3-3a906ecac70e" (UID: "9966eba6-1242-4af5-9fc3-3a906ecac70e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:04:22.155808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:22.155733    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9966eba6-1242-4af5-9fc3-3a906ecac70e-kube-api-access-vr5gx" pod "9966eba6-1242-4af5-9fc3-3a906ecac70e" (UID: "9966eba6-1242-4af5-9fc3-3a906ecac70e"). InnerVolumeSpecName "kube-api-access-vr5gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:04:22.254491 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:22.254425    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9966eba6-1242-4af5-9fc3-3a906ecac70e-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:22.254491 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:22.254453    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr5gx\" (UniqueName: \"kubernetes.io/projected/9966eba6-1242-4af5-9fc3-3a906ecac70e-kube-api-access-vr5gx\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:22.766474 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:22.766444    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:22.767812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:22.767788    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:22.768869 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:22.768851    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:04:22.944811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:22.944780    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h"
Nov 03 08:04:22.944811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:22.944800    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h" event={"ID":"9966eba6-1242-4af5-9fc3-3a906ecac70e","Type":"ContainerDied","Data":"97ee61e98e42eb653f7c6e665d02268030e211228b4ae4094dd8211483a1f45e"}
Nov 03 08:04:22.945018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:22.944860    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97ee61e98e42eb653f7c6e665d02268030e211228b4ae4094dd8211483a1f45e"
Nov 03 08:04:22.998817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:22.998781    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:04:26.939219 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:26.939185    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm"]
Nov 03 08:04:26.939744 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:26.939730    5070 cpu_manager.go:414] "containerMap: removing container" podUID="9966eba6-1242-4af5-9fc3-3a906ecac70e" containerName="revoke-keys"
Nov 03 08:04:26.939826 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:26.939746    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9966eba6-1242-4af5-9fc3-3a906ecac70e" containerName="revoke-keys"
Nov 03 08:04:26.939897 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:26.939874    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="9966eba6-1242-4af5-9fc3-3a906ecac70e" containerName="revoke-keys"
Nov 03 08:04:26.944384 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:26.944366    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm"
Nov 03 08:04:26.947540 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:26.947518    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:04:26.947669 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:26.947627    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:04:26.947732 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:26.947679    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:04:26.947811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:26.947775    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:04:26.950324 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:26.950292    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm"]
Nov 03 08:04:27.019108 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:27.019083    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651375f7-9c24-485b-923b-1c451626cb8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc391d91-cjjjm\" (UID: \"651375f7-9c24-485b-923b-1c451626cb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm"
Nov 03 08:04:27.019245 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:27.019131    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651375f7-9c24-485b-923b-1c451626cb8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc391d91-cjjjm\" (UID: \"651375f7-9c24-485b-923b-1c451626cb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm"
Nov 03 08:04:27.019294 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:27.019262    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bjns\" (UniqueName: \"kubernetes.io/projected/651375f7-9c24-485b-923b-1c451626cb8b-kube-api-access-7bjns\") pod \"maas-api-revoke-sub-fc391d91-cjjjm\" (UID: \"651375f7-9c24-485b-923b-1c451626cb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm"
Nov 03 08:04:27.120166 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:27.120141    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651375f7-9c24-485b-923b-1c451626cb8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc391d91-cjjjm\" (UID: \"651375f7-9c24-485b-923b-1c451626cb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm"
Nov 03 08:04:27.120307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:27.120178    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651375f7-9c24-485b-923b-1c451626cb8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc391d91-cjjjm\" (UID: \"651375f7-9c24-485b-923b-1c451626cb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm"
Nov 03 08:04:27.120307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:27.120242    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bjns\" (UniqueName: \"kubernetes.io/projected/651375f7-9c24-485b-923b-1c451626cb8b-kube-api-access-7bjns\") pod \"maas-api-revoke-sub-fc391d91-cjjjm\" (UID: \"651375f7-9c24-485b-923b-1c451626cb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm"
Nov 03 08:04:27.120898 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:27.120875    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651375f7-9c24-485b-923b-1c451626cb8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc391d91-cjjjm\" (UID: \"651375f7-9c24-485b-923b-1c451626cb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm"
Nov 03 08:04:27.122463 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:27.122446    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651375f7-9c24-485b-923b-1c451626cb8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc391d91-cjjjm\" (UID: \"651375f7-9c24-485b-923b-1c451626cb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm"
Nov 03 08:04:27.126878 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:27.126854    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bjns\" (UniqueName: \"kubernetes.io/projected/651375f7-9c24-485b-923b-1c451626cb8b-kube-api-access-7bjns\") pod \"maas-api-revoke-sub-fc391d91-cjjjm\" (UID: \"651375f7-9c24-485b-923b-1c451626cb8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm"
Nov 03 08:04:27.260408 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:27.260331    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm"
Nov 03 08:04:27.532195 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:27.532170    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm"]
Nov 03 08:04:27.533519 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:04:27.533489    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651375f7_9c24_485b_923b_1c451626cb8b.slice/crio-b0364042f7a8769cbdd64407728fab3b7587d2b85b596ac25a0c248bc7110b3a WatchSource:0}: Error finding container b0364042f7a8769cbdd64407728fab3b7587d2b85b596ac25a0c248bc7110b3a: Status 404 returned error can't find the container with id b0364042f7a8769cbdd64407728fab3b7587d2b85b596ac25a0c248bc7110b3a
Nov 03 08:04:27.995195 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:27.995155    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm" event={"ID":"651375f7-9c24-485b-923b-1c451626cb8b","Type":"ContainerStarted","Data":"f763aada32150c95eb2564f029424b6fd00a7f7389b984e19358606b3b183e15"}
Nov 03 08:04:27.995698 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:27.995246    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm" event={"ID":"651375f7-9c24-485b-923b-1c451626cb8b","Type":"ContainerStarted","Data":"b0364042f7a8769cbdd64407728fab3b7587d2b85b596ac25a0c248bc7110b3a"}
Nov 03 08:04:28.012929 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:28.012883    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm" podStartSLOduration=2.012870036 podStartE2EDuration="2.012870036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:04:28.010769977 +0000 UTC m=+666.125233903" watchObservedRunningTime="2026-11-03 08:04:28.012870036 +0000 UTC m=+666.127333967"
Nov 03 08:04:29.003235 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:29.003202    5070 generic.go:356] "Generic (PLEG): container finished" podID="651375f7-9c24-485b-923b-1c451626cb8b" containerID="f763aada32150c95eb2564f029424b6fd00a7f7389b984e19358606b3b183e15" exitCode=0
Nov 03 08:04:29.003658 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:29.003292    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm" event={"ID":"651375f7-9c24-485b-923b-1c451626cb8b","Type":"ContainerDied","Data":"f763aada32150c95eb2564f029424b6fd00a7f7389b984e19358606b3b183e15"}
Nov 03 08:04:30.164188 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.164165    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm"
Nov 03 08:04:30.253837 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.253798    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/651375f7-9c24-485b-923b-1c451626cb8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651375f7-9c24-485b-923b-1c451626cb8b-maas-api-openshift-ca\") pod \"651375f7-9c24-485b-923b-1c451626cb8b\" (UID: \"651375f7-9c24-485b-923b-1c451626cb8b\") "
Nov 03 08:04:30.254028 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.253950    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/651375f7-9c24-485b-923b-1c451626cb8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651375f7-9c24-485b-923b-1c451626cb8b-maas-api-xks-ca\") pod \"651375f7-9c24-485b-923b-1c451626cb8b\" (UID: \"651375f7-9c24-485b-923b-1c451626cb8b\") "
Nov 03 08:04:30.254028 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.253978    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651375f7-9c24-485b-923b-1c451626cb8b-kube-api-access-7bjns\" (UniqueName: \"kubernetes.io/projected/651375f7-9c24-485b-923b-1c451626cb8b-kube-api-access-7bjns\") pod \"651375f7-9c24-485b-923b-1c451626cb8b\" (UID: \"651375f7-9c24-485b-923b-1c451626cb8b\") "
Nov 03 08:04:30.254184 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.254159    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651375f7-9c24-485b-923b-1c451626cb8b-maas-api-openshift-ca" pod "651375f7-9c24-485b-923b-1c451626cb8b" (UID: "651375f7-9c24-485b-923b-1c451626cb8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:04:30.254346 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.254327    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651375f7-9c24-485b-923b-1c451626cb8b-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:30.256096 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.256072    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/651375f7-9c24-485b-923b-1c451626cb8b-maas-api-xks-ca" pod "651375f7-9c24-485b-923b-1c451626cb8b" (UID: "651375f7-9c24-485b-923b-1c451626cb8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:04:30.256096 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.256077    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651375f7-9c24-485b-923b-1c451626cb8b-kube-api-access-7bjns" pod "651375f7-9c24-485b-923b-1c451626cb8b" (UID: "651375f7-9c24-485b-923b-1c451626cb8b"). InnerVolumeSpecName "kube-api-access-7bjns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:04:30.355985 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.355952    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651375f7-9c24-485b-923b-1c451626cb8b-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:30.355985 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.355987    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bjns\" (UniqueName: \"kubernetes.io/projected/651375f7-9c24-485b-923b-1c451626cb8b-kube-api-access-7bjns\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:30.762871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.762840    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"]
Nov 03 08:04:30.763407 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.763392    5070 cpu_manager.go:414] "containerMap: removing container" podUID="651375f7-9c24-485b-923b-1c451626cb8b" containerName="revoke-keys"
Nov 03 08:04:30.763465 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.763408    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651375f7-9c24-485b-923b-1c451626cb8b" containerName="revoke-keys"
Nov 03 08:04:30.763542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.763533    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="651375f7-9c24-485b-923b-1c451626cb8b" containerName="revoke-keys"
Nov 03 08:04:30.766999 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.766983    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.780429 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.780405    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"]
Nov 03 08:04:30.812108 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.812088    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-ffe7a8-openshift-default-dockercfg-cp99l\""
Nov 03 08:04:30.862293 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.862268    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.862400 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.862314    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.862400 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.862335    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e724a396-646c-42bb-9b59-d7ff618e07f8-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.862476 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.862452    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.862510 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.862497    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.862541 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.862517    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.862589 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.862578    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.862679 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.862660    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.862717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.862689    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t5ff\" (UniqueName: \"kubernetes.io/projected/e724a396-646c-42bb-9b59-d7ff618e07f8-kube-api-access-9t5ff\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.963742 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.963716    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.963854 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.963744    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.963854 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.963819    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.963854 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.963841    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.964010 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.963861    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t5ff\" (UniqueName: \"kubernetes.io/projected/e724a396-646c-42bb-9b59-d7ff618e07f8-kube-api-access-9t5ff\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.964010 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.963933    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.964010 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.963985    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.964162 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.964010    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e724a396-646c-42bb-9b59-d7ff618e07f8-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.964162 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.964089    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.964162 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.964146    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.964324 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.964246    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.964405 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.964376    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.964523 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.964453    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.964930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.964908    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e724a396-646c-42bb-9b59-d7ff618e07f8-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.966020 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.966003    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.966378 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.966356    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.970743 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.970717    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:30.970839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:30.970830    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t5ff\" (UniqueName: \"kubernetes.io/projected/e724a396-646c-42bb-9b59-d7ff618e07f8-kube-api-access-9t5ff\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:31.016475 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:31.016425    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm"
Nov 03 08:04:31.016475 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:31.016443    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm" event={"ID":"651375f7-9c24-485b-923b-1c451626cb8b","Type":"ContainerDied","Data":"b0364042f7a8769cbdd64407728fab3b7587d2b85b596ac25a0c248bc7110b3a"}
Nov 03 08:04:31.016585 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:31.016492    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0364042f7a8769cbdd64407728fab3b7587d2b85b596ac25a0c248bc7110b3a"
Nov 03 08:04:31.082062 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:31.082032    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:31.378470 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:31.378444    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"]
Nov 03 08:04:31.379220 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:04:31.379200    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode724a396_646c_42bb_9b59_d7ff618e07f8.slice/crio-79f83c0c38ef05a99da3f3b313953ce6a8f0a1ffeb4ef8b4d6d624558f5e8320 WatchSource:0}: Error finding container 79f83c0c38ef05a99da3f3b313953ce6a8f0a1ffeb4ef8b4d6d624558f5e8320: Status 404 returned error can't find the container with id 79f83c0c38ef05a99da3f3b313953ce6a8f0a1ffeb4ef8b4d6d624558f5e8320
Nov 03 08:04:31.381592 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:31.381557    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:04:31.381726 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:31.381620    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:04:31.381726 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:31.381652    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:04:32.027543 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:32.027508    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv" event={"ID":"e724a396-646c-42bb-9b59-d7ff618e07f8","Type":"ContainerStarted","Data":"59fbd2ff87bf8abbb5f680d7631cc95bfabb11fe11c6a344dd0c4b87f587df33"}
Nov 03 08:04:32.027698 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:32.027566    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv" event={"ID":"e724a396-646c-42bb-9b59-d7ff618e07f8","Type":"ContainerStarted","Data":"79f83c0c38ef05a99da3f3b313953ce6a8f0a1ffeb4ef8b4d6d624558f5e8320"}
Nov 03 08:04:32.042471 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:32.042429    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv" podStartSLOduration=2.042415046 podStartE2EDuration="2.042415046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:04:32.040840667 +0000 UTC m=+670.155304597" watchObservedRunningTime="2026-11-03 08:04:32.042415046 +0000 UTC m=+670.156879018"
Nov 03 08:04:32.084812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:32.084586    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:32.084812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:32.084797    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 08:04:32.085090 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:32.084847    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv" podUID="e724a396-646c-42bb-9b59-d7ff618e07f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 08:04:33.083217 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:33.083139    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 08:04:33.083217 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:33.083204    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv" podUID="e724a396-646c-42bb-9b59-d7ff618e07f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 08:04:34.082651 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:34.082607    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 08:04:34.082847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:34.082660    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv" podUID="e724a396-646c-42bb-9b59-d7ff618e07f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 08:04:35.084339 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:35.083334    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 08:04:35.084339 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:35.083428    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv" podUID="e724a396-646c-42bb-9b59-d7ff618e07f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 08:04:36.083315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:36.083281    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 08:04:36.083497 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:36.083345    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv" podUID="e724a396-646c-42bb-9b59-d7ff618e07f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 08:04:37.082851 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:37.082812    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Nov 03 08:04:37.083343 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:37.082878    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv" podUID="e724a396-646c-42bb-9b59-d7ff618e07f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Nov 03 08:04:38.352458 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:38.352427    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:38.352893 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:38.352858    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:38.353462 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:38.353444    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:04:41.210783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.209881    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17f8a711-stxs7"]
Nov 03 08:04:41.215544 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.215090    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f8a711-stxs7"
Nov 03 08:04:41.224775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.223927    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:04:41.224775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.224011    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:04:41.224775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.224656    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:04:41.224775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.224774    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:04:41.229779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.229594    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17f8a711-stxs7"]
Nov 03 08:04:41.257304 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.257271    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b8576e-74fd-455f-bd5a-801ffa5ea5b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17f8a711-stxs7\" (UID: \"18b8576e-74fd-455f-bd5a-801ffa5ea5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f8a711-stxs7"
Nov 03 08:04:41.257441 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.257337    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b8576e-74fd-455f-bd5a-801ffa5ea5b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17f8a711-stxs7\" (UID: \"18b8576e-74fd-455f-bd5a-801ffa5ea5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f8a711-stxs7"
Nov 03 08:04:41.257441 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.257393    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nltsf\" (UniqueName: \"kubernetes.io/projected/18b8576e-74fd-455f-bd5a-801ffa5ea5b2-kube-api-access-nltsf\") pod \"maas-api-revoke-sub-17f8a711-stxs7\" (UID: \"18b8576e-74fd-455f-bd5a-801ffa5ea5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f8a711-stxs7"
Nov 03 08:04:41.321137 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.321108    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bf6d7d4-2j8lz"]
Nov 03 08:04:41.325238 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.325222    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf6d7d4-2j8lz"
Nov 03 08:04:41.333237 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.333210    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bf6d7d4-2j8lz"]
Nov 03 08:04:41.358585 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.358557    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f985l\" (UniqueName: \"kubernetes.io/projected/ff9f5ebb-778f-4239-9168-c5706d9d2fbe-kube-api-access-f985l\") pod \"maas-api-revoke-sub-4bf6d7d4-2j8lz\" (UID: \"ff9f5ebb-778f-4239-9168-c5706d9d2fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf6d7d4-2j8lz"
Nov 03 08:04:41.358798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.358686    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b8576e-74fd-455f-bd5a-801ffa5ea5b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17f8a711-stxs7\" (UID: \"18b8576e-74fd-455f-bd5a-801ffa5ea5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f8a711-stxs7"
Nov 03 08:04:41.358798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.358714    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9f5ebb-778f-4239-9168-c5706d9d2fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bf6d7d4-2j8lz\" (UID: \"ff9f5ebb-778f-4239-9168-c5706d9d2fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf6d7d4-2j8lz"
Nov 03 08:04:41.358798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.358797    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b8576e-74fd-455f-bd5a-801ffa5ea5b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17f8a711-stxs7\" (UID: \"18b8576e-74fd-455f-bd5a-801ffa5ea5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f8a711-stxs7"
Nov 03 08:04:41.359003 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.358823    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nltsf\" (UniqueName: \"kubernetes.io/projected/18b8576e-74fd-455f-bd5a-801ffa5ea5b2-kube-api-access-nltsf\") pod \"maas-api-revoke-sub-17f8a711-stxs7\" (UID: \"18b8576e-74fd-455f-bd5a-801ffa5ea5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f8a711-stxs7"
Nov 03 08:04:41.359003 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.358850    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9f5ebb-778f-4239-9168-c5706d9d2fbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bf6d7d4-2j8lz\" (UID: \"ff9f5ebb-778f-4239-9168-c5706d9d2fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf6d7d4-2j8lz"
Nov 03 08:04:41.359956 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.359922    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b8576e-74fd-455f-bd5a-801ffa5ea5b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17f8a711-stxs7\" (UID: \"18b8576e-74fd-455f-bd5a-801ffa5ea5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f8a711-stxs7"
Nov 03 08:04:41.361795 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.361773    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b8576e-74fd-455f-bd5a-801ffa5ea5b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17f8a711-stxs7\" (UID: \"18b8576e-74fd-455f-bd5a-801ffa5ea5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f8a711-stxs7"
Nov 03 08:04:41.387934 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.387909    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nltsf\" (UniqueName: \"kubernetes.io/projected/18b8576e-74fd-455f-bd5a-801ffa5ea5b2-kube-api-access-nltsf\") pod \"maas-api-revoke-sub-17f8a711-stxs7\" (UID: \"18b8576e-74fd-455f-bd5a-801ffa5ea5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f8a711-stxs7"
Nov 03 08:04:41.460584 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.460556    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9f5ebb-778f-4239-9168-c5706d9d2fbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bf6d7d4-2j8lz\" (UID: \"ff9f5ebb-778f-4239-9168-c5706d9d2fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf6d7d4-2j8lz"
Nov 03 08:04:41.460706 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.460644    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f985l\" (UniqueName: \"kubernetes.io/projected/ff9f5ebb-778f-4239-9168-c5706d9d2fbe-kube-api-access-f985l\") pod \"maas-api-revoke-sub-4bf6d7d4-2j8lz\" (UID: \"ff9f5ebb-778f-4239-9168-c5706d9d2fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf6d7d4-2j8lz"
Nov 03 08:04:41.460779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.460738    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9f5ebb-778f-4239-9168-c5706d9d2fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bf6d7d4-2j8lz\" (UID: \"ff9f5ebb-778f-4239-9168-c5706d9d2fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf6d7d4-2j8lz"
Nov 03 08:04:41.461364 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.461308    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9f5ebb-778f-4239-9168-c5706d9d2fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4bf6d7d4-2j8lz\" (UID: \"ff9f5ebb-778f-4239-9168-c5706d9d2fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf6d7d4-2j8lz"
Nov 03 08:04:41.462815 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.462794    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9f5ebb-778f-4239-9168-c5706d9d2fbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4bf6d7d4-2j8lz\" (UID: \"ff9f5ebb-778f-4239-9168-c5706d9d2fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf6d7d4-2j8lz"
Nov 03 08:04:41.468035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.468008    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f985l\" (UniqueName: \"kubernetes.io/projected/ff9f5ebb-778f-4239-9168-c5706d9d2fbe-kube-api-access-f985l\") pod \"maas-api-revoke-sub-4bf6d7d4-2j8lz\" (UID: \"ff9f5ebb-778f-4239-9168-c5706d9d2fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf6d7d4-2j8lz"
Nov 03 08:04:41.557028 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.556997    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f8a711-stxs7"
Nov 03 08:04:41.641822 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.641550    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf6d7d4-2j8lz"
Nov 03 08:04:41.840027 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:41.839996    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17f8a711-stxs7"]
Nov 03 08:04:41.840871 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:04:41.840844    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b8576e_74fd_455f_bd5a_801ffa5ea5b2.slice/crio-f5681db25cbd38748d8f23b5d0bb8a2b12710d1b57631f25f03642f4e4d5fc01 WatchSource:0}: Error finding container f5681db25cbd38748d8f23b5d0bb8a2b12710d1b57631f25f03642f4e4d5fc01: Status 404 returned error can't find the container with id f5681db25cbd38748d8f23b5d0bb8a2b12710d1b57631f25f03642f4e4d5fc01
Nov 03 08:04:42.011699 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.011668    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bf6d7d4-2j8lz"]
Nov 03 08:04:42.024785 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:04:42.023909    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9f5ebb_778f_4239_9168_c5706d9d2fbe.slice/crio-8271772e1b969725833df3100ae8c7e58ebe9c8f6818348bb8a49022f08e66fd WatchSource:0}: Error finding container 8271772e1b969725833df3100ae8c7e58ebe9c8f6818348bb8a49022f08e66fd: Status 404 returned error can't find the container with id 8271772e1b969725833df3100ae8c7e58ebe9c8f6818348bb8a49022f08e66fd
Nov 03 08:04:42.120503 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.120465    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf6d7d4-2j8lz" event={"ID":"ff9f5ebb-778f-4239-9168-c5706d9d2fbe","Type":"ContainerStarted","Data":"8271772e1b969725833df3100ae8c7e58ebe9c8f6818348bb8a49022f08e66fd"}
Nov 03 08:04:42.121782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.121741    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f8a711-stxs7" event={"ID":"18b8576e-74fd-455f-bd5a-801ffa5ea5b2","Type":"ContainerStarted","Data":"f5681db25cbd38748d8f23b5d0bb8a2b12710d1b57631f25f03642f4e4d5fc01"}
Nov 03 08:04:42.414731 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.408887    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02be7f4f-trs9t"]
Nov 03 08:04:42.414731 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.414003    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be7f4f-trs9t"
Nov 03 08:04:42.419301 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.419255    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02be7f4f-trs9t"]
Nov 03 08:04:42.478827 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.478791    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df483417-9936-4124-969e-6d1ff66b7e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02be7f4f-trs9t\" (UID: \"df483417-9936-4124-969e-6d1ff66b7e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be7f4f-trs9t"
Nov 03 08:04:42.479070 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.478842    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8f6w\" (UniqueName: \"kubernetes.io/projected/df483417-9936-4124-969e-6d1ff66b7e06-kube-api-access-s8f6w\") pod \"maas-api-revoke-sub-02be7f4f-trs9t\" (UID: \"df483417-9936-4124-969e-6d1ff66b7e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be7f4f-trs9t"
Nov 03 08:04:42.479070 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.478920    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df483417-9936-4124-969e-6d1ff66b7e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02be7f4f-trs9t\" (UID: \"df483417-9936-4124-969e-6d1ff66b7e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be7f4f-trs9t"
Nov 03 08:04:42.580862 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.580827    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df483417-9936-4124-969e-6d1ff66b7e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02be7f4f-trs9t\" (UID: \"df483417-9936-4124-969e-6d1ff66b7e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be7f4f-trs9t"
Nov 03 08:04:42.581026 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.580871    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8f6w\" (UniqueName: \"kubernetes.io/projected/df483417-9936-4124-969e-6d1ff66b7e06-kube-api-access-s8f6w\") pod \"maas-api-revoke-sub-02be7f4f-trs9t\" (UID: \"df483417-9936-4124-969e-6d1ff66b7e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be7f4f-trs9t"
Nov 03 08:04:42.581026 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.580903    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df483417-9936-4124-969e-6d1ff66b7e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02be7f4f-trs9t\" (UID: \"df483417-9936-4124-969e-6d1ff66b7e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be7f4f-trs9t"
Nov 03 08:04:42.581527 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.581503    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df483417-9936-4124-969e-6d1ff66b7e06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02be7f4f-trs9t\" (UID: \"df483417-9936-4124-969e-6d1ff66b7e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be7f4f-trs9t"
Nov 03 08:04:42.581649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.581621    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499797c6-wznjq"]
Nov 03 08:04:42.583489 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.583462    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df483417-9936-4124-969e-6d1ff66b7e06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02be7f4f-trs9t\" (UID: \"df483417-9936-4124-969e-6d1ff66b7e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be7f4f-trs9t"
Nov 03 08:04:42.585768 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.585740    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499797c6-wznjq"
Nov 03 08:04:42.590538 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.590515    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8f6w\" (UniqueName: \"kubernetes.io/projected/df483417-9936-4124-969e-6d1ff66b7e06-kube-api-access-s8f6w\") pod \"maas-api-revoke-sub-02be7f4f-trs9t\" (UID: \"df483417-9936-4124-969e-6d1ff66b7e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be7f4f-trs9t"
Nov 03 08:04:42.602798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.594025    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499797c6-wznjq"]
Nov 03 08:04:42.681650 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.681622    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab80b3c1-db72-4b77-8ac8-f8b8d6294447-maas-api-xks-ca\") pod \"maas-api-revoke-sub-499797c6-wznjq\" (UID: \"ab80b3c1-db72-4b77-8ac8-f8b8d6294447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499797c6-wznjq"
Nov 03 08:04:42.681804 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.681737    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mptg\" (UniqueName: \"kubernetes.io/projected/ab80b3c1-db72-4b77-8ac8-f8b8d6294447-kube-api-access-4mptg\") pod \"maas-api-revoke-sub-499797c6-wznjq\" (UID: \"ab80b3c1-db72-4b77-8ac8-f8b8d6294447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499797c6-wznjq"
Nov 03 08:04:42.681925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.681909    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab80b3c1-db72-4b77-8ac8-f8b8d6294447-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-499797c6-wznjq\" (UID: \"ab80b3c1-db72-4b77-8ac8-f8b8d6294447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499797c6-wznjq"
Nov 03 08:04:42.736424 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.736403    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be7f4f-trs9t"
Nov 03 08:04:42.792468 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.792438    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab80b3c1-db72-4b77-8ac8-f8b8d6294447-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-499797c6-wznjq\" (UID: \"ab80b3c1-db72-4b77-8ac8-f8b8d6294447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499797c6-wznjq"
Nov 03 08:04:42.792596 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.792548    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab80b3c1-db72-4b77-8ac8-f8b8d6294447-maas-api-xks-ca\") pod \"maas-api-revoke-sub-499797c6-wznjq\" (UID: \"ab80b3c1-db72-4b77-8ac8-f8b8d6294447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499797c6-wznjq"
Nov 03 08:04:42.792660 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.792600    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mptg\" (UniqueName: \"kubernetes.io/projected/ab80b3c1-db72-4b77-8ac8-f8b8d6294447-kube-api-access-4mptg\") pod \"maas-api-revoke-sub-499797c6-wznjq\" (UID: \"ab80b3c1-db72-4b77-8ac8-f8b8d6294447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499797c6-wznjq"
Nov 03 08:04:42.793086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.793065    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab80b3c1-db72-4b77-8ac8-f8b8d6294447-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-499797c6-wznjq\" (UID: \"ab80b3c1-db72-4b77-8ac8-f8b8d6294447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499797c6-wznjq"
Nov 03 08:04:42.794687 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.794667    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab80b3c1-db72-4b77-8ac8-f8b8d6294447-maas-api-xks-ca\") pod \"maas-api-revoke-sub-499797c6-wznjq\" (UID: \"ab80b3c1-db72-4b77-8ac8-f8b8d6294447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499797c6-wznjq"
Nov 03 08:04:42.802446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.802418    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mptg\" (UniqueName: \"kubernetes.io/projected/ab80b3c1-db72-4b77-8ac8-f8b8d6294447-kube-api-access-4mptg\") pod \"maas-api-revoke-sub-499797c6-wznjq\" (UID: \"ab80b3c1-db72-4b77-8ac8-f8b8d6294447\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499797c6-wznjq"
Nov 03 08:04:42.926342 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:42.926279    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499797c6-wznjq"
Nov 03 08:04:43.035082 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:43.035053    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02be7f4f-trs9t"]
Nov 03 08:04:43.036869 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:04:43.036840    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf483417_9936_4124_969e_6d1ff66b7e06.slice/crio-134e50d35f977bad1aae128cb2f331b112098f94eff9eb5d861a1a0089eae207 WatchSource:0}: Error finding container 134e50d35f977bad1aae128cb2f331b112098f94eff9eb5d861a1a0089eae207: Status 404 returned error can't find the container with id 134e50d35f977bad1aae128cb2f331b112098f94eff9eb5d861a1a0089eae207
Nov 03 08:04:43.128547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:43.128518    5070 generic.go:356] "Generic (PLEG): container finished" podID="ff9f5ebb-778f-4239-9168-c5706d9d2fbe" containerID="495058cc171927eb9632a6bb8dd928a3aa7cacd6fa99f34a71226695e7184afc" exitCode=0
Nov 03 08:04:43.128663 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:43.128607    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf6d7d4-2j8lz" event={"ID":"ff9f5ebb-778f-4239-9168-c5706d9d2fbe","Type":"ContainerDied","Data":"495058cc171927eb9632a6bb8dd928a3aa7cacd6fa99f34a71226695e7184afc"}
Nov 03 08:04:43.130145 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:43.130125    5070 generic.go:356] "Generic (PLEG): container finished" podID="18b8576e-74fd-455f-bd5a-801ffa5ea5b2" containerID="d1e89152c7c27f59e6a4261ed14cd3ff3022ec3e47a1f5f9d1ce2f7d19d457ad" exitCode=0
Nov 03 08:04:43.130247 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:43.130194    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f8a711-stxs7" event={"ID":"18b8576e-74fd-455f-bd5a-801ffa5ea5b2","Type":"ContainerDied","Data":"d1e89152c7c27f59e6a4261ed14cd3ff3022ec3e47a1f5f9d1ce2f7d19d457ad"}
Nov 03 08:04:43.131301 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:43.131278    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be7f4f-trs9t" event={"ID":"df483417-9936-4124-969e-6d1ff66b7e06","Type":"ContainerStarted","Data":"134e50d35f977bad1aae128cb2f331b112098f94eff9eb5d861a1a0089eae207"}
Nov 03 08:04:43.211692 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:43.211670    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499797c6-wznjq"]
Nov 03 08:04:43.214438 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:04:43.214413    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab80b3c1_db72_4b77_8ac8_f8b8d6294447.slice/crio-b8b06fcb97edd6d9e43445f050334b4476356be290a066bd4ea9134d9bf95178 WatchSource:0}: Error finding container b8b06fcb97edd6d9e43445f050334b4476356be290a066bd4ea9134d9bf95178: Status 404 returned error can't find the container with id b8b06fcb97edd6d9e43445f050334b4476356be290a066bd4ea9134d9bf95178
Nov 03 08:04:44.139931 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.139845    5070 generic.go:356] "Generic (PLEG): container finished" podID="ab80b3c1-db72-4b77-8ac8-f8b8d6294447" containerID="dec8914c335f9e8567b88277e07ba95958ae5cf7af696e9e84de45c2d4d59dd1" exitCode=0
Nov 03 08:04:44.140355 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.139935    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499797c6-wznjq" event={"ID":"ab80b3c1-db72-4b77-8ac8-f8b8d6294447","Type":"ContainerDied","Data":"dec8914c335f9e8567b88277e07ba95958ae5cf7af696e9e84de45c2d4d59dd1"}
Nov 03 08:04:44.140355 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.139999    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499797c6-wznjq" event={"ID":"ab80b3c1-db72-4b77-8ac8-f8b8d6294447","Type":"ContainerStarted","Data":"b8b06fcb97edd6d9e43445f050334b4476356be290a066bd4ea9134d9bf95178"}
Nov 03 08:04:44.141603 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.141578    5070 generic.go:356] "Generic (PLEG): container finished" podID="df483417-9936-4124-969e-6d1ff66b7e06" containerID="143ff4ac037a8e9eafe96b00feb9cdd0b4563cd4e9c68b6b97a9f76891191623" exitCode=0
Nov 03 08:04:44.141727 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.141656    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be7f4f-trs9t" event={"ID":"df483417-9936-4124-969e-6d1ff66b7e06","Type":"ContainerDied","Data":"143ff4ac037a8e9eafe96b00feb9cdd0b4563cd4e9c68b6b97a9f76891191623"}
Nov 03 08:04:44.305897 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.305872    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f8a711-stxs7"
Nov 03 08:04:44.323472 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.323447    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf6d7d4-2j8lz"
Nov 03 08:04:44.412293 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.412234    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18b8576e-74fd-455f-bd5a-801ffa5ea5b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b8576e-74fd-455f-bd5a-801ffa5ea5b2-maas-api-xks-ca\") pod \"18b8576e-74fd-455f-bd5a-801ffa5ea5b2\" (UID: \"18b8576e-74fd-455f-bd5a-801ffa5ea5b2\") "
Nov 03 08:04:44.412419 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.412333    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff9f5ebb-778f-4239-9168-c5706d9d2fbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9f5ebb-778f-4239-9168-c5706d9d2fbe-maas-api-openshift-ca\") pod \"ff9f5ebb-778f-4239-9168-c5706d9d2fbe\" (UID: \"ff9f5ebb-778f-4239-9168-c5706d9d2fbe\") "
Nov 03 08:04:44.412419 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.412359    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18b8576e-74fd-455f-bd5a-801ffa5ea5b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b8576e-74fd-455f-bd5a-801ffa5ea5b2-maas-api-openshift-ca\") pod \"18b8576e-74fd-455f-bd5a-801ffa5ea5b2\" (UID: \"18b8576e-74fd-455f-bd5a-801ffa5ea5b2\") "
Nov 03 08:04:44.412419 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.412390    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff9f5ebb-778f-4239-9168-c5706d9d2fbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9f5ebb-778f-4239-9168-c5706d9d2fbe-maas-api-xks-ca\") pod \"ff9f5ebb-778f-4239-9168-c5706d9d2fbe\" (UID: \"ff9f5ebb-778f-4239-9168-c5706d9d2fbe\") "
Nov 03 08:04:44.412591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.412431    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff9f5ebb-778f-4239-9168-c5706d9d2fbe-kube-api-access-f985l\" (UniqueName: \"kubernetes.io/projected/ff9f5ebb-778f-4239-9168-c5706d9d2fbe-kube-api-access-f985l\") pod \"ff9f5ebb-778f-4239-9168-c5706d9d2fbe\" (UID: \"ff9f5ebb-778f-4239-9168-c5706d9d2fbe\") "
Nov 03 08:04:44.412591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.412472    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18b8576e-74fd-455f-bd5a-801ffa5ea5b2-kube-api-access-nltsf\" (UniqueName: \"kubernetes.io/projected/18b8576e-74fd-455f-bd5a-801ffa5ea5b2-kube-api-access-nltsf\") pod \"18b8576e-74fd-455f-bd5a-801ffa5ea5b2\" (UID: \"18b8576e-74fd-455f-bd5a-801ffa5ea5b2\") "
Nov 03 08:04:44.412776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.412724    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff9f5ebb-778f-4239-9168-c5706d9d2fbe-maas-api-openshift-ca" pod "ff9f5ebb-778f-4239-9168-c5706d9d2fbe" (UID: "ff9f5ebb-778f-4239-9168-c5706d9d2fbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:04:44.412859 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.412737    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b8576e-74fd-455f-bd5a-801ffa5ea5b2-maas-api-openshift-ca" pod "18b8576e-74fd-455f-bd5a-801ffa5ea5b2" (UID: "18b8576e-74fd-455f-bd5a-801ffa5ea5b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:04:44.413110 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.413084    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9f5ebb-778f-4239-9168-c5706d9d2fbe-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:44.413209 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.413114    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b8576e-74fd-455f-bd5a-801ffa5ea5b2-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:44.414581 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.414552    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b8576e-74fd-455f-bd5a-801ffa5ea5b2-kube-api-access-nltsf" pod "18b8576e-74fd-455f-bd5a-801ffa5ea5b2" (UID: "18b8576e-74fd-455f-bd5a-801ffa5ea5b2"). InnerVolumeSpecName "kube-api-access-nltsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:04:44.414662 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.414636    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff9f5ebb-778f-4239-9168-c5706d9d2fbe-kube-api-access-f985l" pod "ff9f5ebb-778f-4239-9168-c5706d9d2fbe" (UID: "ff9f5ebb-778f-4239-9168-c5706d9d2fbe"). InnerVolumeSpecName "kube-api-access-f985l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:04:44.414871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.414848    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9f5ebb-778f-4239-9168-c5706d9d2fbe-maas-api-xks-ca" pod "ff9f5ebb-778f-4239-9168-c5706d9d2fbe" (UID: "ff9f5ebb-778f-4239-9168-c5706d9d2fbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:04:44.414921 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.414872    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18b8576e-74fd-455f-bd5a-801ffa5ea5b2-maas-api-xks-ca" pod "18b8576e-74fd-455f-bd5a-801ffa5ea5b2" (UID: "18b8576e-74fd-455f-bd5a-801ffa5ea5b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:04:44.513735 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.513707    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9f5ebb-778f-4239-9168-c5706d9d2fbe-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:44.513853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.513783    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f985l\" (UniqueName: \"kubernetes.io/projected/ff9f5ebb-778f-4239-9168-c5706d9d2fbe-kube-api-access-f985l\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:44.513853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.513797    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nltsf\" (UniqueName: \"kubernetes.io/projected/18b8576e-74fd-455f-bd5a-801ffa5ea5b2-kube-api-access-nltsf\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:44.513853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:44.513807    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b8576e-74fd-455f-bd5a-801ffa5ea5b2-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:45.149870 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.149841    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf6d7d4-2j8lz"
Nov 03 08:04:45.150306 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.149871    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4bf6d7d4-2j8lz" event={"ID":"ff9f5ebb-778f-4239-9168-c5706d9d2fbe","Type":"ContainerDied","Data":"8271772e1b969725833df3100ae8c7e58ebe9c8f6818348bb8a49022f08e66fd"}
Nov 03 08:04:45.150306 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.149919    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8271772e1b969725833df3100ae8c7e58ebe9c8f6818348bb8a49022f08e66fd"
Nov 03 08:04:45.151271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.151254    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f8a711-stxs7"
Nov 03 08:04:45.151391 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.151293    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17f8a711-stxs7" event={"ID":"18b8576e-74fd-455f-bd5a-801ffa5ea5b2","Type":"ContainerDied","Data":"f5681db25cbd38748d8f23b5d0bb8a2b12710d1b57631f25f03642f4e4d5fc01"}
Nov 03 08:04:45.151391 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.151326    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5681db25cbd38748d8f23b5d0bb8a2b12710d1b57631f25f03642f4e4d5fc01"
Nov 03 08:04:45.342903 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.342878    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be7f4f-trs9t"
Nov 03 08:04:45.344825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.344808    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499797c6-wznjq"
Nov 03 08:04:45.423120 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.423049    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab80b3c1-db72-4b77-8ac8-f8b8d6294447-kube-api-access-4mptg\" (UniqueName: \"kubernetes.io/projected/ab80b3c1-db72-4b77-8ac8-f8b8d6294447-kube-api-access-4mptg\") pod \"ab80b3c1-db72-4b77-8ac8-f8b8d6294447\" (UID: \"ab80b3c1-db72-4b77-8ac8-f8b8d6294447\") "
Nov 03 08:04:45.423120 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.423116    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df483417-9936-4124-969e-6d1ff66b7e06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df483417-9936-4124-969e-6d1ff66b7e06-maas-api-openshift-ca\") pod \"df483417-9936-4124-969e-6d1ff66b7e06\" (UID: \"df483417-9936-4124-969e-6d1ff66b7e06\") "
Nov 03 08:04:45.423325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.423247    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df483417-9936-4124-969e-6d1ff66b7e06-kube-api-access-s8f6w\" (UniqueName: \"kubernetes.io/projected/df483417-9936-4124-969e-6d1ff66b7e06-kube-api-access-s8f6w\") pod \"df483417-9936-4124-969e-6d1ff66b7e06\" (UID: \"df483417-9936-4124-969e-6d1ff66b7e06\") "
Nov 03 08:04:45.423325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.423277    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab80b3c1-db72-4b77-8ac8-f8b8d6294447-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab80b3c1-db72-4b77-8ac8-f8b8d6294447-maas-api-openshift-ca\") pod \"ab80b3c1-db72-4b77-8ac8-f8b8d6294447\" (UID: \"ab80b3c1-db72-4b77-8ac8-f8b8d6294447\") "
Nov 03 08:04:45.423466 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.423442    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df483417-9936-4124-969e-6d1ff66b7e06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df483417-9936-4124-969e-6d1ff66b7e06-maas-api-xks-ca\") pod \"df483417-9936-4124-969e-6d1ff66b7e06\" (UID: \"df483417-9936-4124-969e-6d1ff66b7e06\") "
Nov 03 08:04:45.424130 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.423491    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab80b3c1-db72-4b77-8ac8-f8b8d6294447-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab80b3c1-db72-4b77-8ac8-f8b8d6294447-maas-api-xks-ca\") pod \"ab80b3c1-db72-4b77-8ac8-f8b8d6294447\" (UID: \"ab80b3c1-db72-4b77-8ac8-f8b8d6294447\") "
Nov 03 08:04:45.424952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.424925    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df483417-9936-4124-969e-6d1ff66b7e06-maas-api-openshift-ca" pod "df483417-9936-4124-969e-6d1ff66b7e06" (UID: "df483417-9936-4124-969e-6d1ff66b7e06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:04:45.425151 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.425079    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab80b3c1-db72-4b77-8ac8-f8b8d6294447-maas-api-openshift-ca" pod "ab80b3c1-db72-4b77-8ac8-f8b8d6294447" (UID: "ab80b3c1-db72-4b77-8ac8-f8b8d6294447"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:04:45.427705 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.427684    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab80b3c1-db72-4b77-8ac8-f8b8d6294447-maas-api-xks-ca" pod "ab80b3c1-db72-4b77-8ac8-f8b8d6294447" (UID: "ab80b3c1-db72-4b77-8ac8-f8b8d6294447"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:04:45.427809 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.427698    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df483417-9936-4124-969e-6d1ff66b7e06-kube-api-access-s8f6w" pod "df483417-9936-4124-969e-6d1ff66b7e06" (UID: "df483417-9936-4124-969e-6d1ff66b7e06"). InnerVolumeSpecName "kube-api-access-s8f6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:04:45.428601 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.428580    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df483417-9936-4124-969e-6d1ff66b7e06-maas-api-xks-ca" pod "df483417-9936-4124-969e-6d1ff66b7e06" (UID: "df483417-9936-4124-969e-6d1ff66b7e06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:04:45.428932 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.428909    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab80b3c1-db72-4b77-8ac8-f8b8d6294447-kube-api-access-4mptg" pod "ab80b3c1-db72-4b77-8ac8-f8b8d6294447" (UID: "ab80b3c1-db72-4b77-8ac8-f8b8d6294447"). InnerVolumeSpecName "kube-api-access-4mptg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:04:45.521419 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.521387    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"]
Nov 03 08:04:45.521955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.521937    5070 cpu_manager.go:414] "containerMap: removing container" podUID="ff9f5ebb-778f-4239-9168-c5706d9d2fbe" containerName="revoke-keys"
Nov 03 08:04:45.521955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.521952    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff9f5ebb-778f-4239-9168-c5706d9d2fbe" containerName="revoke-keys"
Nov 03 08:04:45.522115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.521970    5070 cpu_manager.go:414] "containerMap: removing container" podUID="18b8576e-74fd-455f-bd5a-801ffa5ea5b2" containerName="revoke-keys"
Nov 03 08:04:45.522115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.521975    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18b8576e-74fd-455f-bd5a-801ffa5ea5b2" containerName="revoke-keys"
Nov 03 08:04:45.522115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.521988    5070 cpu_manager.go:414] "containerMap: removing container" podUID="ab80b3c1-db72-4b77-8ac8-f8b8d6294447" containerName="revoke-keys"
Nov 03 08:04:45.522115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.521994    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab80b3c1-db72-4b77-8ac8-f8b8d6294447" containerName="revoke-keys"
Nov 03 08:04:45.522115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.522007    5070 cpu_manager.go:414] "containerMap: removing container" podUID="df483417-9936-4124-969e-6d1ff66b7e06" containerName="revoke-keys"
Nov 03 08:04:45.522115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.522011    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df483417-9936-4124-969e-6d1ff66b7e06" containerName="revoke-keys"
Nov 03 08:04:45.522115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.522094    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="18b8576e-74fd-455f-bd5a-801ffa5ea5b2" containerName="revoke-keys"
Nov 03 08:04:45.522115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.522107    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab80b3c1-db72-4b77-8ac8-f8b8d6294447" containerName="revoke-keys"
Nov 03 08:04:45.522115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.522118    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="df483417-9936-4124-969e-6d1ff66b7e06" containerName="revoke-keys"
Nov 03 08:04:45.522541 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.522131    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff9f5ebb-778f-4239-9168-c5706d9d2fbe" containerName="revoke-keys"
Nov 03 08:04:45.525493 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.525469    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df483417-9936-4124-969e-6d1ff66b7e06-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:45.525493 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.525491    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab80b3c1-db72-4b77-8ac8-f8b8d6294447-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:45.525655 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.525500    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mptg\" (UniqueName: \"kubernetes.io/projected/ab80b3c1-db72-4b77-8ac8-f8b8d6294447-kube-api-access-4mptg\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:45.525655 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.525509    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df483417-9936-4124-969e-6d1ff66b7e06-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:45.525655 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.525518    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8f6w\" (UniqueName: \"kubernetes.io/projected/df483417-9936-4124-969e-6d1ff66b7e06-kube-api-access-s8f6w\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:45.525655 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.525526    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab80b3c1-db72-4b77-8ac8-f8b8d6294447-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:45.525879 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.525725    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.537687 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.537663    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"]
Nov 03 08:04:45.629160 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.629102    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.629447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.629176    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhk94\" (UniqueName: \"kubernetes.io/projected/2abf1628-8c68-450c-a458-eefa78b239b3-kube-api-access-fhk94\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.629532 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.629427    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.629639 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.629595    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.629639 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.629617    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abf1628-8c68-450c-a458-eefa78b239b3-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.629783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.629734    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.629888 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.629873    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.629928 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.629920    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abf1628-8c68-450c-a458-eefa78b239b3-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.630087 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.630050    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abf1628-8c68-450c-a458-eefa78b239b3-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.731800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.731691    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abf1628-8c68-450c-a458-eefa78b239b3-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.731800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.731745    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.732006 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.731800    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhk94\" (UniqueName: \"kubernetes.io/projected/2abf1628-8c68-450c-a458-eefa78b239b3-kube-api-access-fhk94\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.732006 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.731847    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.732006 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.731889    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.732006 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.731904    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abf1628-8c68-450c-a458-eefa78b239b3-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.732006 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.731958    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.732337 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.732007    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.732337 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.732061    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abf1628-8c68-450c-a458-eefa78b239b3-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.732337 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.732184    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.732337 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.732259    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.732569 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.732476    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abf1628-8c68-450c-a458-eefa78b239b3-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.732652 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.732630    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.732652 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.732640    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.734252 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.734230    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.734508 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.734486    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abf1628-8c68-450c-a458-eefa78b239b3-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.738670 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.738648    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abf1628-8c68-450c-a458-eefa78b239b3-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.738745 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.738716    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhk94\" (UniqueName: \"kubernetes.io/projected/2abf1628-8c68-450c-a458-eefa78b239b3-kube-api-access-fhk94\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:45.847040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:45.847017    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:46.128283 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:46.128257    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"]
Nov 03 08:04:46.129151 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:04:46.129127    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abf1628_8c68_450c_a458_eefa78b239b3.slice/crio-6cdce6e38e494579f85911a169a6b02f68662d54ad1c4c78be24bb2794097def WatchSource:0}: Error finding container 6cdce6e38e494579f85911a169a6b02f68662d54ad1c4c78be24bb2794097def: Status 404 returned error can't find the container with id 6cdce6e38e494579f85911a169a6b02f68662d54ad1c4c78be24bb2794097def
Nov 03 08:04:46.131388 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:46.131345    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:04:46.131477 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:46.131417    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:04:46.131477 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:46.131467    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:04:46.161901 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:46.161881    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499797c6-wznjq"
Nov 03 08:04:46.162236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:46.161908    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499797c6-wznjq" event={"ID":"ab80b3c1-db72-4b77-8ac8-f8b8d6294447","Type":"ContainerDied","Data":"b8b06fcb97edd6d9e43445f050334b4476356be290a066bd4ea9134d9bf95178"}
Nov 03 08:04:46.162236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:46.161961    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8b06fcb97edd6d9e43445f050334b4476356be290a066bd4ea9134d9bf95178"
Nov 03 08:04:46.163629 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:46.163583    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs" event={"ID":"2abf1628-8c68-450c-a458-eefa78b239b3","Type":"ContainerStarted","Data":"6cdce6e38e494579f85911a169a6b02f68662d54ad1c4c78be24bb2794097def"}
Nov 03 08:04:46.165187 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:46.165167    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be7f4f-trs9t"
Nov 03 08:04:46.165307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:46.165194    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02be7f4f-trs9t" event={"ID":"df483417-9936-4124-969e-6d1ff66b7e06","Type":"ContainerDied","Data":"134e50d35f977bad1aae128cb2f331b112098f94eff9eb5d861a1a0089eae207"}
Nov 03 08:04:46.165307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:46.165233    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="134e50d35f977bad1aae128cb2f331b112098f94eff9eb5d861a1a0089eae207"
Nov 03 08:04:47.189144 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:47.189108    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs" event={"ID":"2abf1628-8c68-450c-a458-eefa78b239b3","Type":"ContainerStarted","Data":"d93bb40e960d77605b81abd3b9d587c894c99bcfd66e65d1133c662cfa9e242b"}
Nov 03 08:04:47.207609 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:47.207562    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs" podStartSLOduration=2.207549777 podStartE2EDuration="2.207549777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:04:47.20544061 +0000 UTC m=+685.319904541" watchObservedRunningTime="2026-11-03 08:04:47.207549777 +0000 UTC m=+685.322013708"
Nov 03 08:04:47.847601 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:47.847571    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:47.850798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:47.849862    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 03 08:04:47.850798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:47.849927    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs" podUID="2abf1628-8c68-450c-a458-eefa78b239b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 03 08:04:48.850783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:48.849941    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 03 08:04:48.850783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:48.850015    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs" podUID="2abf1628-8c68-450c-a458-eefa78b239b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 03 08:04:49.847780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:49.847729    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 03 08:04:49.847982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:49.847831    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs" podUID="2abf1628-8c68-450c-a458-eefa78b239b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 03 08:04:50.847538 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:50.847500    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 03 08:04:50.847965 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:50.847559    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs" podUID="2abf1628-8c68-450c-a458-eefa78b239b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 03 08:04:51.847955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:51.847920    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 03 08:04:51.848364 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:51.847970    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs" podUID="2abf1628-8c68-450c-a458-eefa78b239b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 03 08:04:52.050400 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:04:52.050362    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:04:52.632222 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:52.632187    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1b36786-2742x"]
Nov 03 08:04:52.640842 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:52.640817    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b36786-2742x"
Nov 03 08:04:52.647612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:52.645748    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:04:52.647612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:52.645886    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:04:52.647612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:52.645952    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:04:52.647612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:52.646146    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:04:52.647612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:52.646916    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1b36786-2742x"]
Nov 03 08:04:52.814341 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:52.814299    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkm4r\" (UniqueName: \"kubernetes.io/projected/5d21bbe7-804b-4e67-93fc-94e90ce47a9f-kube-api-access-jkm4r\") pod \"maas-api-revoke-sub-a1b36786-2742x\" (UID: \"5d21bbe7-804b-4e67-93fc-94e90ce47a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b36786-2742x"
Nov 03 08:04:52.814530 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:52.814418    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d21bbe7-804b-4e67-93fc-94e90ce47a9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1b36786-2742x\" (UID: \"5d21bbe7-804b-4e67-93fc-94e90ce47a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b36786-2742x"
Nov 03 08:04:52.814530 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:52.814449    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d21bbe7-804b-4e67-93fc-94e90ce47a9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1b36786-2742x\" (UID: \"5d21bbe7-804b-4e67-93fc-94e90ce47a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b36786-2742x"
Nov 03 08:04:52.847572 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:52.847544    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 03 08:04:52.847727 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:52.847593    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs" podUID="2abf1628-8c68-450c-a458-eefa78b239b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 03 08:04:52.916202 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:52.916121    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkm4r\" (UniqueName: \"kubernetes.io/projected/5d21bbe7-804b-4e67-93fc-94e90ce47a9f-kube-api-access-jkm4r\") pod \"maas-api-revoke-sub-a1b36786-2742x\" (UID: \"5d21bbe7-804b-4e67-93fc-94e90ce47a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b36786-2742x"
Nov 03 08:04:52.916642 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:52.916221    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d21bbe7-804b-4e67-93fc-94e90ce47a9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1b36786-2742x\" (UID: \"5d21bbe7-804b-4e67-93fc-94e90ce47a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b36786-2742x"
Nov 03 08:04:52.916642 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:52.916245    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d21bbe7-804b-4e67-93fc-94e90ce47a9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1b36786-2742x\" (UID: \"5d21bbe7-804b-4e67-93fc-94e90ce47a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b36786-2742x"
Nov 03 08:04:52.916966 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:52.916944    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d21bbe7-804b-4e67-93fc-94e90ce47a9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1b36786-2742x\" (UID: \"5d21bbe7-804b-4e67-93fc-94e90ce47a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b36786-2742x"
Nov 03 08:04:52.918808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:52.918743    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d21bbe7-804b-4e67-93fc-94e90ce47a9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1b36786-2742x\" (UID: \"5d21bbe7-804b-4e67-93fc-94e90ce47a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b36786-2742x"
Nov 03 08:04:52.923289 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:52.923264    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkm4r\" (UniqueName: \"kubernetes.io/projected/5d21bbe7-804b-4e67-93fc-94e90ce47a9f-kube-api-access-jkm4r\") pod \"maas-api-revoke-sub-a1b36786-2742x\" (UID: \"5d21bbe7-804b-4e67-93fc-94e90ce47a9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b36786-2742x"
Nov 03 08:04:52.974423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:52.974395    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b36786-2742x"
Nov 03 08:04:53.041572 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.041535    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp"]
Nov 03 08:04:53.055258 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.054448    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp"]
Nov 03 08:04:53.061063 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.058413    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp"
Nov 03 08:04:53.119899 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.119859    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac3a802-ff04-4427-9be4-d517a61495fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b28de902-fg2jp\" (UID: \"8ac3a802-ff04-4427-9be4-d517a61495fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp"
Nov 03 08:04:53.120071 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.119906    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tll5c\" (UniqueName: \"kubernetes.io/projected/8ac3a802-ff04-4427-9be4-d517a61495fc-kube-api-access-tll5c\") pod \"maas-api-revoke-sub-b28de902-fg2jp\" (UID: \"8ac3a802-ff04-4427-9be4-d517a61495fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp"
Nov 03 08:04:53.120071 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.119932    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac3a802-ff04-4427-9be4-d517a61495fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b28de902-fg2jp\" (UID: \"8ac3a802-ff04-4427-9be4-d517a61495fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp"
Nov 03 08:04:53.143067 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.143028    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-393ef7cc-6qkps"]
Nov 03 08:04:53.147543 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.147523    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-393ef7cc-6qkps"
Nov 03 08:04:53.158013 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.156807    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-393ef7cc-6qkps"]
Nov 03 08:04:53.222437 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.222351    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69a19836-dd98-4498-bc17-86d681f1b39e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-393ef7cc-6qkps\" (UID: \"69a19836-dd98-4498-bc17-86d681f1b39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-393ef7cc-6qkps"
Nov 03 08:04:53.222900 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.222874    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f8bq\" (UniqueName: \"kubernetes.io/projected/69a19836-dd98-4498-bc17-86d681f1b39e-kube-api-access-7f8bq\") pod \"maas-api-revoke-sub-393ef7cc-6qkps\" (UID: \"69a19836-dd98-4498-bc17-86d681f1b39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-393ef7cc-6qkps"
Nov 03 08:04:53.223164 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.223144    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69a19836-dd98-4498-bc17-86d681f1b39e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-393ef7cc-6qkps\" (UID: \"69a19836-dd98-4498-bc17-86d681f1b39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-393ef7cc-6qkps"
Nov 03 08:04:53.223370 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.223332    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac3a802-ff04-4427-9be4-d517a61495fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b28de902-fg2jp\" (UID: \"8ac3a802-ff04-4427-9be4-d517a61495fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp"
Nov 03 08:04:53.223523 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.223508    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tll5c\" (UniqueName: \"kubernetes.io/projected/8ac3a802-ff04-4427-9be4-d517a61495fc-kube-api-access-tll5c\") pod \"maas-api-revoke-sub-b28de902-fg2jp\" (UID: \"8ac3a802-ff04-4427-9be4-d517a61495fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp"
Nov 03 08:04:53.223674 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.223659    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac3a802-ff04-4427-9be4-d517a61495fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b28de902-fg2jp\" (UID: \"8ac3a802-ff04-4427-9be4-d517a61495fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp"
Nov 03 08:04:53.224575 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.224549    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac3a802-ff04-4427-9be4-d517a61495fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b28de902-fg2jp\" (UID: \"8ac3a802-ff04-4427-9be4-d517a61495fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp"
Nov 03 08:04:53.226645 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.226620    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac3a802-ff04-4427-9be4-d517a61495fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b28de902-fg2jp\" (UID: \"8ac3a802-ff04-4427-9be4-d517a61495fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp"
Nov 03 08:04:53.230972 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.230952    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tll5c\" (UniqueName: \"kubernetes.io/projected/8ac3a802-ff04-4427-9be4-d517a61495fc-kube-api-access-tll5c\") pod \"maas-api-revoke-sub-b28de902-fg2jp\" (UID: \"8ac3a802-ff04-4427-9be4-d517a61495fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp"
Nov 03 08:04:53.296006 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.295956    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1b36786-2742x"]
Nov 03 08:04:53.321179 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.321143    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b36786-2742x" event={"ID":"5d21bbe7-804b-4e67-93fc-94e90ce47a9f","Type":"ContainerStarted","Data":"3ea38e108fc33f301a4a5291859cc476ccc6c4f398ce9f1d51bead9460cd5dc8"}
Nov 03 08:04:53.325112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.325083    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f8bq\" (UniqueName: \"kubernetes.io/projected/69a19836-dd98-4498-bc17-86d681f1b39e-kube-api-access-7f8bq\") pod \"maas-api-revoke-sub-393ef7cc-6qkps\" (UID: \"69a19836-dd98-4498-bc17-86d681f1b39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-393ef7cc-6qkps"
Nov 03 08:04:53.325395 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.325366    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69a19836-dd98-4498-bc17-86d681f1b39e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-393ef7cc-6qkps\" (UID: \"69a19836-dd98-4498-bc17-86d681f1b39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-393ef7cc-6qkps"
Nov 03 08:04:53.325514 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.325470    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69a19836-dd98-4498-bc17-86d681f1b39e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-393ef7cc-6qkps\" (UID: \"69a19836-dd98-4498-bc17-86d681f1b39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-393ef7cc-6qkps"
Nov 03 08:04:53.326092 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.326071    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69a19836-dd98-4498-bc17-86d681f1b39e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-393ef7cc-6qkps\" (UID: \"69a19836-dd98-4498-bc17-86d681f1b39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-393ef7cc-6qkps"
Nov 03 08:04:53.327875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.327855    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69a19836-dd98-4498-bc17-86d681f1b39e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-393ef7cc-6qkps\" (UID: \"69a19836-dd98-4498-bc17-86d681f1b39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-393ef7cc-6qkps"
Nov 03 08:04:53.332922 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.332893    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f8bq\" (UniqueName: \"kubernetes.io/projected/69a19836-dd98-4498-bc17-86d681f1b39e-kube-api-access-7f8bq\") pod \"maas-api-revoke-sub-393ef7cc-6qkps\" (UID: \"69a19836-dd98-4498-bc17-86d681f1b39e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-393ef7cc-6qkps"
Nov 03 08:04:53.381073 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.381055    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp"
Nov 03 08:04:53.424023 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.423990    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe0ba50-t52nr"]
Nov 03 08:04:53.429507 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.429486    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0ba50-t52nr"
Nov 03 08:04:53.435881 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.435857    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe0ba50-t52nr"]
Nov 03 08:04:53.468784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.468496    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-393ef7cc-6qkps"
Nov 03 08:04:53.528526 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.528481    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw7xs\" (UniqueName: \"kubernetes.io/projected/6104730c-9250-4000-a32b-a13dd6128ce0-kube-api-access-nw7xs\") pod \"maas-api-revoke-sub-1fe0ba50-t52nr\" (UID: \"6104730c-9250-4000-a32b-a13dd6128ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0ba50-t52nr"
Nov 03 08:04:53.528693 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.528676    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6104730c-9250-4000-a32b-a13dd6128ce0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fe0ba50-t52nr\" (UID: \"6104730c-9250-4000-a32b-a13dd6128ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0ba50-t52nr"
Nov 03 08:04:53.528864 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.528730    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6104730c-9250-4000-a32b-a13dd6128ce0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fe0ba50-t52nr\" (UID: \"6104730c-9250-4000-a32b-a13dd6128ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0ba50-t52nr"
Nov 03 08:04:53.630190 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.630162    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw7xs\" (UniqueName: \"kubernetes.io/projected/6104730c-9250-4000-a32b-a13dd6128ce0-kube-api-access-nw7xs\") pod \"maas-api-revoke-sub-1fe0ba50-t52nr\" (UID: \"6104730c-9250-4000-a32b-a13dd6128ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0ba50-t52nr"
Nov 03 08:04:53.630355 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.630314    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6104730c-9250-4000-a32b-a13dd6128ce0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fe0ba50-t52nr\" (UID: \"6104730c-9250-4000-a32b-a13dd6128ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0ba50-t52nr"
Nov 03 08:04:53.630475 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.630459    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6104730c-9250-4000-a32b-a13dd6128ce0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fe0ba50-t52nr\" (UID: \"6104730c-9250-4000-a32b-a13dd6128ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0ba50-t52nr"
Nov 03 08:04:53.631077 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.631048    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6104730c-9250-4000-a32b-a13dd6128ce0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fe0ba50-t52nr\" (UID: \"6104730c-9250-4000-a32b-a13dd6128ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0ba50-t52nr"
Nov 03 08:04:53.633057 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.633036    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6104730c-9250-4000-a32b-a13dd6128ce0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fe0ba50-t52nr\" (UID: \"6104730c-9250-4000-a32b-a13dd6128ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0ba50-t52nr"
Nov 03 08:04:53.636605 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.636584    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw7xs\" (UniqueName: \"kubernetes.io/projected/6104730c-9250-4000-a32b-a13dd6128ce0-kube-api-access-nw7xs\") pod \"maas-api-revoke-sub-1fe0ba50-t52nr\" (UID: \"6104730c-9250-4000-a32b-a13dd6128ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0ba50-t52nr"
Nov 03 08:04:53.670719 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.670660    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp"]
Nov 03 08:04:53.672629 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:04:53.672608    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac3a802_ff04_4427_9be4_d517a61495fc.slice/crio-3981b03652efd2862f161b7fdeafe7786c38718d6e341371b10efe7df5777605 WatchSource:0}: Error finding container 3981b03652efd2862f161b7fdeafe7786c38718d6e341371b10efe7df5777605: Status 404 returned error can't find the container with id 3981b03652efd2862f161b7fdeafe7786c38718d6e341371b10efe7df5777605
Nov 03 08:04:53.752991 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.752923    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0ba50-t52nr"
Nov 03 08:04:53.776934 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:53.776893    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-393ef7cc-6qkps"]
Nov 03 08:04:54.150530 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:54.150498    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe0ba50-t52nr"]
Nov 03 08:04:54.151013 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:04:54.150947    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6104730c_9250_4000_a32b_a13dd6128ce0.slice/crio-4966deb227b85a20d791db603aada3c27c6b7d0beabdaa514bf3f934ad55e59f WatchSource:0}: Error finding container 4966deb227b85a20d791db603aada3c27c6b7d0beabdaa514bf3f934ad55e59f: Status 404 returned error can't find the container with id 4966deb227b85a20d791db603aada3c27c6b7d0beabdaa514bf3f934ad55e59f
Nov 03 08:04:54.222352 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:54.222326    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:54.222876 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:54.222848    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:54.223800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:54.223749    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:04:54.329977 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:54.329939    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp" event={"ID":"8ac3a802-ff04-4427-9be4-d517a61495fc","Type":"ContainerStarted","Data":"00ba44e6f4690abe1a92b3cacf20792fe0d2b6775b6480938a3ae15f1ea43c9b"}
Nov 03 08:04:54.330138 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:54.330004    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp" event={"ID":"8ac3a802-ff04-4427-9be4-d517a61495fc","Type":"ContainerStarted","Data":"3981b03652efd2862f161b7fdeafe7786c38718d6e341371b10efe7df5777605"}
Nov 03 08:04:54.331401 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:54.331369    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0ba50-t52nr" event={"ID":"6104730c-9250-4000-a32b-a13dd6128ce0","Type":"ContainerStarted","Data":"4966deb227b85a20d791db603aada3c27c6b7d0beabdaa514bf3f934ad55e59f"}
Nov 03 08:04:54.332964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:54.332940    5070 generic.go:356] "Generic (PLEG): container finished" podID="5d21bbe7-804b-4e67-93fc-94e90ce47a9f" containerID="3aa707899708a2ef8d335fad977ffb0ee0e467dd4aa4c991f9774a13bbaf71f3" exitCode=0
Nov 03 08:04:54.333066 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:54.332997    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b36786-2742x" event={"ID":"5d21bbe7-804b-4e67-93fc-94e90ce47a9f","Type":"ContainerDied","Data":"3aa707899708a2ef8d335fad977ffb0ee0e467dd4aa4c991f9774a13bbaf71f3"}
Nov 03 08:04:54.334403 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:54.334381    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-393ef7cc-6qkps" event={"ID":"69a19836-dd98-4498-bc17-86d681f1b39e","Type":"ContainerStarted","Data":"3f2c832e0367853f961cc485423df74664e2da76bae3d41e064475edca241837"}
Nov 03 08:04:54.346368 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:54.346322    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp" podStartSLOduration=1.346306217 podStartE2EDuration="1.346306217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:04:54.345315426 +0000 UTC m=+692.459779357" watchObservedRunningTime="2026-11-03 08:04:54.346306217 +0000 UTC m=+692.460770148"
Nov 03 08:04:55.342061 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.342030    5070 generic.go:356] "Generic (PLEG): container finished" podID="8ac3a802-ff04-4427-9be4-d517a61495fc" containerID="00ba44e6f4690abe1a92b3cacf20792fe0d2b6775b6480938a3ae15f1ea43c9b" exitCode=0
Nov 03 08:04:55.342399 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.342116    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp" event={"ID":"8ac3a802-ff04-4427-9be4-d517a61495fc","Type":"ContainerDied","Data":"00ba44e6f4690abe1a92b3cacf20792fe0d2b6775b6480938a3ae15f1ea43c9b"}
Nov 03 08:04:55.343564 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.343538    5070 generic.go:356] "Generic (PLEG): container finished" podID="6104730c-9250-4000-a32b-a13dd6128ce0" containerID="e43182984345f5f147d9fc3dfe43c833e1b9a7db8c3f9854624b5babfa8ffc3c" exitCode=0
Nov 03 08:04:55.343683 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.343626    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0ba50-t52nr" event={"ID":"6104730c-9250-4000-a32b-a13dd6128ce0","Type":"ContainerDied","Data":"e43182984345f5f147d9fc3dfe43c833e1b9a7db8c3f9854624b5babfa8ffc3c"}
Nov 03 08:04:55.345536 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.345511    5070 generic.go:356] "Generic (PLEG): container finished" podID="69a19836-dd98-4498-bc17-86d681f1b39e" containerID="bb5ace5e28fb545b53a02d6e38b3007d317de568b9e88e5ef225ef916b2647fe" exitCode=0
Nov 03 08:04:55.346168 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.346148    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-393ef7cc-6qkps" event={"ID":"69a19836-dd98-4498-bc17-86d681f1b39e","Type":"ContainerDied","Data":"bb5ace5e28fb545b53a02d6e38b3007d317de568b9e88e5ef225ef916b2647fe"}
Nov 03 08:04:55.519829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.519807    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b36786-2742x"
Nov 03 08:04:55.552082 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.552051    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d21bbe7-804b-4e67-93fc-94e90ce47a9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d21bbe7-804b-4e67-93fc-94e90ce47a9f-maas-api-openshift-ca\") pod \"5d21bbe7-804b-4e67-93fc-94e90ce47a9f\" (UID: \"5d21bbe7-804b-4e67-93fc-94e90ce47a9f\") "
Nov 03 08:04:55.552236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.552148    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d21bbe7-804b-4e67-93fc-94e90ce47a9f-kube-api-access-jkm4r\" (UniqueName: \"kubernetes.io/projected/5d21bbe7-804b-4e67-93fc-94e90ce47a9f-kube-api-access-jkm4r\") pod \"5d21bbe7-804b-4e67-93fc-94e90ce47a9f\" (UID: \"5d21bbe7-804b-4e67-93fc-94e90ce47a9f\") "
Nov 03 08:04:55.552293 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.552240    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d21bbe7-804b-4e67-93fc-94e90ce47a9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d21bbe7-804b-4e67-93fc-94e90ce47a9f-maas-api-xks-ca\") pod \"5d21bbe7-804b-4e67-93fc-94e90ce47a9f\" (UID: \"5d21bbe7-804b-4e67-93fc-94e90ce47a9f\") "
Nov 03 08:04:55.553647 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.553614    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d21bbe7-804b-4e67-93fc-94e90ce47a9f-maas-api-openshift-ca" pod "5d21bbe7-804b-4e67-93fc-94e90ce47a9f" (UID: "5d21bbe7-804b-4e67-93fc-94e90ce47a9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:04:55.557880 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.557855    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d21bbe7-804b-4e67-93fc-94e90ce47a9f-maas-api-xks-ca" pod "5d21bbe7-804b-4e67-93fc-94e90ce47a9f" (UID: "5d21bbe7-804b-4e67-93fc-94e90ce47a9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:04:55.558033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.557914    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d21bbe7-804b-4e67-93fc-94e90ce47a9f-kube-api-access-jkm4r" pod "5d21bbe7-804b-4e67-93fc-94e90ce47a9f" (UID: "5d21bbe7-804b-4e67-93fc-94e90ce47a9f"). InnerVolumeSpecName "kube-api-access-jkm4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:04:55.654345 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.654316    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkm4r\" (UniqueName: \"kubernetes.io/projected/5d21bbe7-804b-4e67-93fc-94e90ce47a9f-kube-api-access-jkm4r\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:55.654345 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.654345    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d21bbe7-804b-4e67-93fc-94e90ce47a9f-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:55.654576 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.654355    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d21bbe7-804b-4e67-93fc-94e90ce47a9f-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:55.801550 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.801470    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq"]
Nov 03 08:04:55.802173 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.802116    5070 cpu_manager.go:414] "containerMap: removing container" podUID="5d21bbe7-804b-4e67-93fc-94e90ce47a9f" containerName="revoke-keys"
Nov 03 08:04:55.802173 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.802138    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d21bbe7-804b-4e67-93fc-94e90ce47a9f" containerName="revoke-keys"
Nov 03 08:04:55.802354 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.802236    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d21bbe7-804b-4e67-93fc-94e90ce47a9f" containerName="revoke-keys"
Nov 03 08:04:55.805868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.805850    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq"
Nov 03 08:04:55.812991 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.812965    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq"]
Nov 03 08:04:55.857049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.857017    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfdnp\" (UniqueName: \"kubernetes.io/projected/85feb58c-6d48-4e9f-96e4-0273fc44a9eb-kube-api-access-jfdnp\") pod \"maas-api-revoke-sub-a71a5e70-45xsq\" (UID: \"85feb58c-6d48-4e9f-96e4-0273fc44a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq"
Nov 03 08:04:55.857227 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.857207    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85feb58c-6d48-4e9f-96e4-0273fc44a9eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a71a5e70-45xsq\" (UID: \"85feb58c-6d48-4e9f-96e4-0273fc44a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq"
Nov 03 08:04:55.857424 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.857401    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85feb58c-6d48-4e9f-96e4-0273fc44a9eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a71a5e70-45xsq\" (UID: \"85feb58c-6d48-4e9f-96e4-0273fc44a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq"
Nov 03 08:04:55.958630 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.958590    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85feb58c-6d48-4e9f-96e4-0273fc44a9eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a71a5e70-45xsq\" (UID: \"85feb58c-6d48-4e9f-96e4-0273fc44a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq"
Nov 03 08:04:55.958840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.958663    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85feb58c-6d48-4e9f-96e4-0273fc44a9eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a71a5e70-45xsq\" (UID: \"85feb58c-6d48-4e9f-96e4-0273fc44a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq"
Nov 03 08:04:55.958840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.958694    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfdnp\" (UniqueName: \"kubernetes.io/projected/85feb58c-6d48-4e9f-96e4-0273fc44a9eb-kube-api-access-jfdnp\") pod \"maas-api-revoke-sub-a71a5e70-45xsq\" (UID: \"85feb58c-6d48-4e9f-96e4-0273fc44a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq"
Nov 03 08:04:55.959282 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.959260    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85feb58c-6d48-4e9f-96e4-0273fc44a9eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a71a5e70-45xsq\" (UID: \"85feb58c-6d48-4e9f-96e4-0273fc44a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq"
Nov 03 08:04:55.961045 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.961021    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85feb58c-6d48-4e9f-96e4-0273fc44a9eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a71a5e70-45xsq\" (UID: \"85feb58c-6d48-4e9f-96e4-0273fc44a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq"
Nov 03 08:04:55.969587 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:55.969566    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfdnp\" (UniqueName: \"kubernetes.io/projected/85feb58c-6d48-4e9f-96e4-0273fc44a9eb-kube-api-access-jfdnp\") pod \"maas-api-revoke-sub-a71a5e70-45xsq\" (UID: \"85feb58c-6d48-4e9f-96e4-0273fc44a9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq"
Nov 03 08:04:56.122294 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.122272    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq"
Nov 03 08:04:56.354066 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.354035    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b36786-2742x"
Nov 03 08:04:56.354482 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.354072    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1b36786-2742x" event={"ID":"5d21bbe7-804b-4e67-93fc-94e90ce47a9f","Type":"ContainerDied","Data":"3ea38e108fc33f301a4a5291859cc476ccc6c4f398ce9f1d51bead9460cd5dc8"}
Nov 03 08:04:56.354482 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.354113    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ea38e108fc33f301a4a5291859cc476ccc6c4f398ce9f1d51bead9460cd5dc8"
Nov 03 08:04:56.399441 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.399408    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq"]
Nov 03 08:04:56.399898 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:04:56.399874    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85feb58c_6d48_4e9f_96e4_0273fc44a9eb.slice/crio-a3165cef0005bdc4efe0a16490eecfc5882c061b36794516eba49ec9536a308e WatchSource:0}: Error finding container a3165cef0005bdc4efe0a16490eecfc5882c061b36794516eba49ec9536a308e: Status 404 returned error can't find the container with id a3165cef0005bdc4efe0a16490eecfc5882c061b36794516eba49ec9536a308e
Nov 03 08:04:56.505681 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.505650    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0ba50-t52nr"
Nov 03 08:04:56.566770 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.566730    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6104730c-9250-4000-a32b-a13dd6128ce0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6104730c-9250-4000-a32b-a13dd6128ce0-maas-api-openshift-ca\") pod \"6104730c-9250-4000-a32b-a13dd6128ce0\" (UID: \"6104730c-9250-4000-a32b-a13dd6128ce0\") "
Nov 03 08:04:56.567022 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.567000    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6104730c-9250-4000-a32b-a13dd6128ce0-kube-api-access-nw7xs\" (UniqueName: \"kubernetes.io/projected/6104730c-9250-4000-a32b-a13dd6128ce0-kube-api-access-nw7xs\") pod \"6104730c-9250-4000-a32b-a13dd6128ce0\" (UID: \"6104730c-9250-4000-a32b-a13dd6128ce0\") "
Nov 03 08:04:56.567073 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.567050    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6104730c-9250-4000-a32b-a13dd6128ce0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6104730c-9250-4000-a32b-a13dd6128ce0-maas-api-xks-ca\") pod \"6104730c-9250-4000-a32b-a13dd6128ce0\" (UID: \"6104730c-9250-4000-a32b-a13dd6128ce0\") "
Nov 03 08:04:56.567145 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.567073    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6104730c-9250-4000-a32b-a13dd6128ce0-maas-api-openshift-ca" pod "6104730c-9250-4000-a32b-a13dd6128ce0" (UID: "6104730c-9250-4000-a32b-a13dd6128ce0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:04:56.567458 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.567436    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6104730c-9250-4000-a32b-a13dd6128ce0-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:56.569549 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.569337    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6104730c-9250-4000-a32b-a13dd6128ce0-maas-api-xks-ca" pod "6104730c-9250-4000-a32b-a13dd6128ce0" (UID: "6104730c-9250-4000-a32b-a13dd6128ce0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:04:56.569971 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.569938    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6104730c-9250-4000-a32b-a13dd6128ce0-kube-api-access-nw7xs" pod "6104730c-9250-4000-a32b-a13dd6128ce0" (UID: "6104730c-9250-4000-a32b-a13dd6128ce0"). InnerVolumeSpecName "kube-api-access-nw7xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:04:56.583965 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.583946    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp"
Nov 03 08:04:56.587199 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.587183    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-393ef7cc-6qkps"
Nov 03 08:04:56.667950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.667875    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69a19836-dd98-4498-bc17-86d681f1b39e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69a19836-dd98-4498-bc17-86d681f1b39e-maas-api-xks-ca\") pod \"69a19836-dd98-4498-bc17-86d681f1b39e\" (UID: \"69a19836-dd98-4498-bc17-86d681f1b39e\") "
Nov 03 08:04:56.668100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.667996    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ac3a802-ff04-4427-9be4-d517a61495fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac3a802-ff04-4427-9be4-d517a61495fc-maas-api-openshift-ca\") pod \"8ac3a802-ff04-4427-9be4-d517a61495fc\" (UID: \"8ac3a802-ff04-4427-9be4-d517a61495fc\") "
Nov 03 08:04:56.668100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.668081    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69a19836-dd98-4498-bc17-86d681f1b39e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69a19836-dd98-4498-bc17-86d681f1b39e-maas-api-openshift-ca\") pod \"69a19836-dd98-4498-bc17-86d681f1b39e\" (UID: \"69a19836-dd98-4498-bc17-86d681f1b39e\") "
Nov 03 08:04:56.668226 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.668146    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ac3a802-ff04-4427-9be4-d517a61495fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac3a802-ff04-4427-9be4-d517a61495fc-maas-api-xks-ca\") pod \"8ac3a802-ff04-4427-9be4-d517a61495fc\" (UID: \"8ac3a802-ff04-4427-9be4-d517a61495fc\") "
Nov 03 08:04:56.668226 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.668205    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a19836-dd98-4498-bc17-86d681f1b39e-kube-api-access-7f8bq\" (UniqueName: \"kubernetes.io/projected/69a19836-dd98-4498-bc17-86d681f1b39e-kube-api-access-7f8bq\") pod \"69a19836-dd98-4498-bc17-86d681f1b39e\" (UID: \"69a19836-dd98-4498-bc17-86d681f1b39e\") "
Nov 03 08:04:56.668439 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.668289    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac3a802-ff04-4427-9be4-d517a61495fc-kube-api-access-tll5c\" (UniqueName: \"kubernetes.io/projected/8ac3a802-ff04-4427-9be4-d517a61495fc-kube-api-access-tll5c\") pod \"8ac3a802-ff04-4427-9be4-d517a61495fc\" (UID: \"8ac3a802-ff04-4427-9be4-d517a61495fc\") "
Nov 03 08:04:56.668621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.668576    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac3a802-ff04-4427-9be4-d517a61495fc-maas-api-openshift-ca" pod "8ac3a802-ff04-4427-9be4-d517a61495fc" (UID: "8ac3a802-ff04-4427-9be4-d517a61495fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:04:56.668910 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.668889    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw7xs\" (UniqueName: \"kubernetes.io/projected/6104730c-9250-4000-a32b-a13dd6128ce0-kube-api-access-nw7xs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:56.669073 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.668917    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6104730c-9250-4000-a32b-a13dd6128ce0-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:56.669073 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.668933    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ac3a802-ff04-4427-9be4-d517a61495fc-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:56.669315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.669176    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a19836-dd98-4498-bc17-86d681f1b39e-maas-api-openshift-ca" pod "69a19836-dd98-4498-bc17-86d681f1b39e" (UID: "69a19836-dd98-4498-bc17-86d681f1b39e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:04:56.670740 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.670708    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69a19836-dd98-4498-bc17-86d681f1b39e-maas-api-xks-ca" pod "69a19836-dd98-4498-bc17-86d681f1b39e" (UID: "69a19836-dd98-4498-bc17-86d681f1b39e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:04:56.671185 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.671160    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a19836-dd98-4498-bc17-86d681f1b39e-kube-api-access-7f8bq" pod "69a19836-dd98-4498-bc17-86d681f1b39e" (UID: "69a19836-dd98-4498-bc17-86d681f1b39e"). InnerVolumeSpecName "kube-api-access-7f8bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:04:56.671596 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.671573    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac3a802-ff04-4427-9be4-d517a61495fc-kube-api-access-tll5c" pod "8ac3a802-ff04-4427-9be4-d517a61495fc" (UID: "8ac3a802-ff04-4427-9be4-d517a61495fc"). InnerVolumeSpecName "kube-api-access-tll5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:04:56.672031 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.672009    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac3a802-ff04-4427-9be4-d517a61495fc-maas-api-xks-ca" pod "8ac3a802-ff04-4427-9be4-d517a61495fc" (UID: "8ac3a802-ff04-4427-9be4-d517a61495fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:04:56.770605 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.770572    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69a19836-dd98-4498-bc17-86d681f1b39e-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:56.770722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.770610    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ac3a802-ff04-4427-9be4-d517a61495fc-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:56.770722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.770625    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f8bq\" (UniqueName: \"kubernetes.io/projected/69a19836-dd98-4498-bc17-86d681f1b39e-kube-api-access-7f8bq\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:56.770722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.770640    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tll5c\" (UniqueName: \"kubernetes.io/projected/8ac3a802-ff04-4427-9be4-d517a61495fc-kube-api-access-tll5c\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:56.770722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:56.770653    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69a19836-dd98-4498-bc17-86d681f1b39e-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:57.362667 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:57.362643    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-393ef7cc-6qkps"
Nov 03 08:04:57.362667 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:57.362651    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-393ef7cc-6qkps" event={"ID":"69a19836-dd98-4498-bc17-86d681f1b39e","Type":"ContainerDied","Data":"3f2c832e0367853f961cc485423df74664e2da76bae3d41e064475edca241837"}
Nov 03 08:04:57.363114 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:57.362703    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f2c832e0367853f961cc485423df74664e2da76bae3d41e064475edca241837"
Nov 03 08:04:57.365192 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:57.365165    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq" event={"ID":"85feb58c-6d48-4e9f-96e4-0273fc44a9eb","Type":"ContainerStarted","Data":"0ebfec6bb88f79329e61eb5a0c009df297b40c4118285b2490134a2061052eae"}
Nov 03 08:04:57.365327 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:57.365219    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq" event={"ID":"85feb58c-6d48-4e9f-96e4-0273fc44a9eb","Type":"ContainerStarted","Data":"a3165cef0005bdc4efe0a16490eecfc5882c061b36794516eba49ec9536a308e"}
Nov 03 08:04:57.366706 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:57.366690    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp"
Nov 03 08:04:57.366845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:57.366710    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp" event={"ID":"8ac3a802-ff04-4427-9be4-d517a61495fc","Type":"ContainerDied","Data":"3981b03652efd2862f161b7fdeafe7786c38718d6e341371b10efe7df5777605"}
Nov 03 08:04:57.366845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:57.366752    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3981b03652efd2862f161b7fdeafe7786c38718d6e341371b10efe7df5777605"
Nov 03 08:04:57.368391 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:57.368366    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0ba50-t52nr"
Nov 03 08:04:57.368644 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:57.368388    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0ba50-t52nr" event={"ID":"6104730c-9250-4000-a32b-a13dd6128ce0","Type":"ContainerDied","Data":"4966deb227b85a20d791db603aada3c27c6b7d0beabdaa514bf3f934ad55e59f"}
Nov 03 08:04:57.368752 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:57.368657    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4966deb227b85a20d791db603aada3c27c6b7d0beabdaa514bf3f934ad55e59f"
Nov 03 08:04:57.389320 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:57.389281    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq" podStartSLOduration=2.389271175 podStartE2EDuration="2.389271175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:04:57.387592633 +0000 UTC m=+695.502056565" watchObservedRunningTime="2026-11-03 08:04:57.389271175 +0000 UTC m=+695.503735145"
Nov 03 08:04:58.382528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:58.379829    5070 generic.go:356] "Generic (PLEG): container finished" podID="85feb58c-6d48-4e9f-96e4-0273fc44a9eb" containerID="0ebfec6bb88f79329e61eb5a0c009df297b40c4118285b2490134a2061052eae" exitCode=0
Nov 03 08:04:58.382528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:58.379871    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq" event={"ID":"85feb58c-6d48-4e9f-96e4-0273fc44a9eb","Type":"ContainerDied","Data":"0ebfec6bb88f79329e61eb5a0c009df297b40c4118285b2490134a2061052eae"}
Nov 03 08:04:59.544875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:59.544849    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq"
Nov 03 08:04:59.605217 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:59.605189    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85feb58c-6d48-4e9f-96e4-0273fc44a9eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85feb58c-6d48-4e9f-96e4-0273fc44a9eb-maas-api-xks-ca\") pod \"85feb58c-6d48-4e9f-96e4-0273fc44a9eb\" (UID: \"85feb58c-6d48-4e9f-96e4-0273fc44a9eb\") "
Nov 03 08:04:59.605396 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:59.605383    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85feb58c-6d48-4e9f-96e4-0273fc44a9eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85feb58c-6d48-4e9f-96e4-0273fc44a9eb-maas-api-openshift-ca\") pod \"85feb58c-6d48-4e9f-96e4-0273fc44a9eb\" (UID: \"85feb58c-6d48-4e9f-96e4-0273fc44a9eb\") "
Nov 03 08:04:59.605459 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:59.605420    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85feb58c-6d48-4e9f-96e4-0273fc44a9eb-kube-api-access-jfdnp\" (UniqueName: \"kubernetes.io/projected/85feb58c-6d48-4e9f-96e4-0273fc44a9eb-kube-api-access-jfdnp\") pod \"85feb58c-6d48-4e9f-96e4-0273fc44a9eb\" (UID: \"85feb58c-6d48-4e9f-96e4-0273fc44a9eb\") "
Nov 03 08:04:59.605790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:59.605749    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85feb58c-6d48-4e9f-96e4-0273fc44a9eb-maas-api-openshift-ca" pod "85feb58c-6d48-4e9f-96e4-0273fc44a9eb" (UID: "85feb58c-6d48-4e9f-96e4-0273fc44a9eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:04:59.607199 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:59.607173    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85feb58c-6d48-4e9f-96e4-0273fc44a9eb-maas-api-xks-ca" pod "85feb58c-6d48-4e9f-96e4-0273fc44a9eb" (UID: "85feb58c-6d48-4e9f-96e4-0273fc44a9eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:04:59.607558 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:59.607537    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85feb58c-6d48-4e9f-96e4-0273fc44a9eb-kube-api-access-jfdnp" pod "85feb58c-6d48-4e9f-96e4-0273fc44a9eb" (UID: "85feb58c-6d48-4e9f-96e4-0273fc44a9eb"). InnerVolumeSpecName "kube-api-access-jfdnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:04:59.707311 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:59.707248    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85feb58c-6d48-4e9f-96e4-0273fc44a9eb-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:59.707311 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:59.707271    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85feb58c-6d48-4e9f-96e4-0273fc44a9eb-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:04:59.707311 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:04:59.707282    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfdnp\" (UniqueName: \"kubernetes.io/projected/85feb58c-6d48-4e9f-96e4-0273fc44a9eb-kube-api-access-jfdnp\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:00.383404 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:00.383373    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:05:00.393977 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:00.393954    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq"
Nov 03 08:05:00.394144 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:00.393998    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq" event={"ID":"85feb58c-6d48-4e9f-96e4-0273fc44a9eb","Type":"ContainerDied","Data":"a3165cef0005bdc4efe0a16490eecfc5882c061b36794516eba49ec9536a308e"}
Nov 03 08:05:00.394144 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:00.394042    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3165cef0005bdc4efe0a16490eecfc5882c061b36794516eba49ec9536a308e"
Nov 03 08:05:02.559897 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:02.559849    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:05:02.590808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:02.577824    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:02.595891 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:02.595319    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:02.671780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:02.670874    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:05:02.673357 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:02.673312    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:02.820005 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:02.819888    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:05.545942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.545912    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n"]
Nov 03 08:05:05.546448 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.546436    5070 cpu_manager.go:414] "containerMap: removing container" podUID="85feb58c-6d48-4e9f-96e4-0273fc44a9eb" containerName="revoke-keys"
Nov 03 08:05:05.546496 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.546450    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85feb58c-6d48-4e9f-96e4-0273fc44a9eb" containerName="revoke-keys"
Nov 03 08:05:05.546496 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.546460    5070 cpu_manager.go:414] "containerMap: removing container" podUID="8ac3a802-ff04-4427-9be4-d517a61495fc" containerName="revoke-keys"
Nov 03 08:05:05.546496 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.546465    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac3a802-ff04-4427-9be4-d517a61495fc" containerName="revoke-keys"
Nov 03 08:05:05.546496 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.546482    5070 cpu_manager.go:414] "containerMap: removing container" podUID="69a19836-dd98-4498-bc17-86d681f1b39e" containerName="revoke-keys"
Nov 03 08:05:05.546496 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.546487    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69a19836-dd98-4498-bc17-86d681f1b39e" containerName="revoke-keys"
Nov 03 08:05:05.546689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.546500    5070 cpu_manager.go:414] "containerMap: removing container" podUID="6104730c-9250-4000-a32b-a13dd6128ce0" containerName="revoke-keys"
Nov 03 08:05:05.546689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.546505    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6104730c-9250-4000-a32b-a13dd6128ce0" containerName="revoke-keys"
Nov 03 08:05:05.546689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.546592    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac3a802-ff04-4427-9be4-d517a61495fc" containerName="revoke-keys"
Nov 03 08:05:05.546689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.546601    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="69a19836-dd98-4498-bc17-86d681f1b39e" containerName="revoke-keys"
Nov 03 08:05:05.546689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.546608    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="6104730c-9250-4000-a32b-a13dd6128ce0" containerName="revoke-keys"
Nov 03 08:05:05.546689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.546619    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="85feb58c-6d48-4e9f-96e4-0273fc44a9eb" containerName="revoke-keys"
Nov 03 08:05:05.551226 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.551208    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n"
Nov 03 08:05:05.556379 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.556355    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:05:05.556379 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.556355    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:05:05.556565 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.556345    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:05:05.556750 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.556433    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:05:05.557240 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.557218    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n"]
Nov 03 08:05:05.638123 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.637540    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaec62f7-ghj6n\" (UID: \"3fbc395a-05ae-4c54-a6a6-acdbc8e29f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n"
Nov 03 08:05:05.638123 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.637635    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r7xj\" (UniqueName: \"kubernetes.io/projected/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00-kube-api-access-7r7xj\") pod \"maas-api-revoke-sub-aaec62f7-ghj6n\" (UID: \"3fbc395a-05ae-4c54-a6a6-acdbc8e29f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n"
Nov 03 08:05:05.638123 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.637728    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaec62f7-ghj6n\" (UID: \"3fbc395a-05ae-4c54-a6a6-acdbc8e29f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n"
Nov 03 08:05:05.738463 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.738431    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaec62f7-ghj6n\" (UID: \"3fbc395a-05ae-4c54-a6a6-acdbc8e29f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n"
Nov 03 08:05:05.738642 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.738515    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r7xj\" (UniqueName: \"kubernetes.io/projected/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00-kube-api-access-7r7xj\") pod \"maas-api-revoke-sub-aaec62f7-ghj6n\" (UID: \"3fbc395a-05ae-4c54-a6a6-acdbc8e29f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n"
Nov 03 08:05:05.738642 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.738604    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaec62f7-ghj6n\" (UID: \"3fbc395a-05ae-4c54-a6a6-acdbc8e29f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n"
Nov 03 08:05:05.739271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.739242    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaec62f7-ghj6n\" (UID: \"3fbc395a-05ae-4c54-a6a6-acdbc8e29f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n"
Nov 03 08:05:05.740856 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.740833    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaec62f7-ghj6n\" (UID: \"3fbc395a-05ae-4c54-a6a6-acdbc8e29f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n"
Nov 03 08:05:05.745252 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.745225    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r7xj\" (UniqueName: \"kubernetes.io/projected/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00-kube-api-access-7r7xj\") pod \"maas-api-revoke-sub-aaec62f7-ghj6n\" (UID: \"3fbc395a-05ae-4c54-a6a6-acdbc8e29f00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n"
Nov 03 08:05:05.866200 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:05.866173    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n"
Nov 03 08:05:06.001057 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.001026    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e33e4b-tm2n4"]
Nov 03 08:05:06.027021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.026989    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e33e4b-tm2n4"
Nov 03 08:05:06.039384 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.039343    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e33e4b-tm2n4"]
Nov 03 08:05:06.041319 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.041252    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4e33e4b-tm2n4\" (UID: \"9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e33e4b-tm2n4"
Nov 03 08:05:06.041436 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.041334    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8df5w\" (UniqueName: \"kubernetes.io/projected/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3-kube-api-access-8df5w\") pod \"maas-api-revoke-sub-b4e33e4b-tm2n4\" (UID: \"9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e33e4b-tm2n4"
Nov 03 08:05:06.041436 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.041424    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4e33e4b-tm2n4\" (UID: \"9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e33e4b-tm2n4"
Nov 03 08:05:06.142610 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.142554    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4e33e4b-tm2n4\" (UID: \"9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e33e4b-tm2n4"
Nov 03 08:05:06.142610 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.142592    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8df5w\" (UniqueName: \"kubernetes.io/projected/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3-kube-api-access-8df5w\") pod \"maas-api-revoke-sub-b4e33e4b-tm2n4\" (UID: \"9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e33e4b-tm2n4"
Nov 03 08:05:06.142772 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.142652    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4e33e4b-tm2n4\" (UID: \"9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e33e4b-tm2n4"
Nov 03 08:05:06.143233 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.143209    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4e33e4b-tm2n4\" (UID: \"9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e33e4b-tm2n4"
Nov 03 08:05:06.144987 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.144967    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4e33e4b-tm2n4\" (UID: \"9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e33e4b-tm2n4"
Nov 03 08:05:06.151379 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.151353    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8df5w\" (UniqueName: \"kubernetes.io/projected/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3-kube-api-access-8df5w\") pod \"maas-api-revoke-sub-b4e33e4b-tm2n4\" (UID: \"9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e33e4b-tm2n4"
Nov 03 08:05:06.156787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.156739    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n"]
Nov 03 08:05:06.157838 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:05:06.157815    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbc395a_05ae_4c54_a6a6_acdbc8e29f00.slice/crio-acc17bdb60193777110fc13abac3d391330e9e5468743808a496dff3912c15b7 WatchSource:0}: Error finding container acc17bdb60193777110fc13abac3d391330e9e5468743808a496dff3912c15b7: Status 404 returned error can't find the container with id acc17bdb60193777110fc13abac3d391330e9e5468743808a496dff3912c15b7
Nov 03 08:05:06.276700 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.276675    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbef5506-hm49w"]
Nov 03 08:05:06.281896 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.281881    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbef5506-hm49w"
Nov 03 08:05:06.288183 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.288153    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbef5506-hm49w"]
Nov 03 08:05:06.345171 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.345146    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-642kc\" (UniqueName: \"kubernetes.io/projected/7db497ac-2392-4e2d-9b56-0398213ced6c-kube-api-access-642kc\") pod \"maas-api-revoke-sub-fbef5506-hm49w\" (UID: \"7db497ac-2392-4e2d-9b56-0398213ced6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbef5506-hm49w"
Nov 03 08:05:06.345333 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.345191    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db497ac-2392-4e2d-9b56-0398213ced6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbef5506-hm49w\" (UID: \"7db497ac-2392-4e2d-9b56-0398213ced6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbef5506-hm49w"
Nov 03 08:05:06.345333 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.345324    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db497ac-2392-4e2d-9b56-0398213ced6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbef5506-hm49w\" (UID: \"7db497ac-2392-4e2d-9b56-0398213ced6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbef5506-hm49w"
Nov 03 08:05:06.356875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.356853    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e33e4b-tm2n4"
Nov 03 08:05:06.446955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.446922    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-642kc\" (UniqueName: \"kubernetes.io/projected/7db497ac-2392-4e2d-9b56-0398213ced6c-kube-api-access-642kc\") pod \"maas-api-revoke-sub-fbef5506-hm49w\" (UID: \"7db497ac-2392-4e2d-9b56-0398213ced6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbef5506-hm49w"
Nov 03 08:05:06.447076 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.446983    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db497ac-2392-4e2d-9b56-0398213ced6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbef5506-hm49w\" (UID: \"7db497ac-2392-4e2d-9b56-0398213ced6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbef5506-hm49w"
Nov 03 08:05:06.447076 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.447061    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db497ac-2392-4e2d-9b56-0398213ced6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbef5506-hm49w\" (UID: \"7db497ac-2392-4e2d-9b56-0398213ced6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbef5506-hm49w"
Nov 03 08:05:06.455139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.447832    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db497ac-2392-4e2d-9b56-0398213ced6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbef5506-hm49w\" (UID: \"7db497ac-2392-4e2d-9b56-0398213ced6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbef5506-hm49w"
Nov 03 08:05:06.455139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.449714    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db497ac-2392-4e2d-9b56-0398213ced6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbef5506-hm49w\" (UID: \"7db497ac-2392-4e2d-9b56-0398213ced6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbef5506-hm49w"
Nov 03 08:05:06.455718 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.455668    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-642kc\" (UniqueName: \"kubernetes.io/projected/7db497ac-2392-4e2d-9b56-0398213ced6c-kube-api-access-642kc\") pod \"maas-api-revoke-sub-fbef5506-hm49w\" (UID: \"7db497ac-2392-4e2d-9b56-0398213ced6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbef5506-hm49w"
Nov 03 08:05:06.600072 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.600037    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbef5506-hm49w"
Nov 03 08:05:06.633258 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.633200    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e33e4b-tm2n4"]
Nov 03 08:05:06.633955 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:05:06.633925    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed4d3f4_88e6_47a0_a3b9_906b570fa3c3.slice/crio-db8c45acf6c7227ac8bb8a43b3eb8cdd6e82916aeecc6519cd6dbd3322cfd4f6 WatchSource:0}: Error finding container db8c45acf6c7227ac8bb8a43b3eb8cdd6e82916aeecc6519cd6dbd3322cfd4f6: Status 404 returned error can't find the container with id db8c45acf6c7227ac8bb8a43b3eb8cdd6e82916aeecc6519cd6dbd3322cfd4f6
Nov 03 08:05:06.684625 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.675702    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n" event={"ID":"3fbc395a-05ae-4c54-a6a6-acdbc8e29f00","Type":"ContainerStarted","Data":"ded35c498d485f28149a61be7fcd1767503ce1885bc0005d40c1ab726179a0d4"}
Nov 03 08:05:06.684625 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.675797    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n" event={"ID":"3fbc395a-05ae-4c54-a6a6-acdbc8e29f00","Type":"ContainerStarted","Data":"acc17bdb60193777110fc13abac3d391330e9e5468743808a496dff3912c15b7"}
Nov 03 08:05:06.684625 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.680312    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e33e4b-tm2n4" event={"ID":"9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3","Type":"ContainerStarted","Data":"db8c45acf6c7227ac8bb8a43b3eb8cdd6e82916aeecc6519cd6dbd3322cfd4f6"}
Nov 03 08:05:06.693541 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.693480    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n" podStartSLOduration=1.693470008 podStartE2EDuration="1.693470008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:06.693279861 +0000 UTC m=+704.807743793" watchObservedRunningTime="2026-11-03 08:05:06.693470008 +0000 UTC m=+704.807933939"
Nov 03 08:05:06.881807 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:05:06.881780    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7db497ac_2392_4e2d_9b56_0398213ced6c.slice/crio-400f8bb8384b73101196680842e10125a2120ee946fb70ef1fb05ac3044a898a WatchSource:0}: Error finding container 400f8bb8384b73101196680842e10125a2120ee946fb70ef1fb05ac3044a898a: Status 404 returned error can't find the container with id 400f8bb8384b73101196680842e10125a2120ee946fb70ef1fb05ac3044a898a
Nov 03 08:05:06.881807 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:06.881790    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbef5506-hm49w"]
Nov 03 08:05:07.687952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:07.687922    5070 generic.go:356] "Generic (PLEG): container finished" podID="3fbc395a-05ae-4c54-a6a6-acdbc8e29f00" containerID="ded35c498d485f28149a61be7fcd1767503ce1885bc0005d40c1ab726179a0d4" exitCode=0
Nov 03 08:05:07.688271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:07.687968    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n" event={"ID":"3fbc395a-05ae-4c54-a6a6-acdbc8e29f00","Type":"ContainerDied","Data":"ded35c498d485f28149a61be7fcd1767503ce1885bc0005d40c1ab726179a0d4"}
Nov 03 08:05:07.689606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:07.689576    5070 generic.go:356] "Generic (PLEG): container finished" podID="7db497ac-2392-4e2d-9b56-0398213ced6c" containerID="e85d13146ad67eceb3dcbae6604eff49ac6be873d1c3294edca35c63c59e5723" exitCode=0
Nov 03 08:05:07.689700 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:07.689653    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbef5506-hm49w" event={"ID":"7db497ac-2392-4e2d-9b56-0398213ced6c","Type":"ContainerDied","Data":"e85d13146ad67eceb3dcbae6604eff49ac6be873d1c3294edca35c63c59e5723"}
Nov 03 08:05:07.689777 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:07.689712    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbef5506-hm49w" event={"ID":"7db497ac-2392-4e2d-9b56-0398213ced6c","Type":"ContainerStarted","Data":"400f8bb8384b73101196680842e10125a2120ee946fb70ef1fb05ac3044a898a"}
Nov 03 08:05:07.691225 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:07.691204    5070 generic.go:356] "Generic (PLEG): container finished" podID="9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3" containerID="27755c39299093d7bf79a29651e39c3c497060438cb6b14cf5c9154b62f4790c" exitCode=0
Nov 03 08:05:07.691322 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:07.691261    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e33e4b-tm2n4" event={"ID":"9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3","Type":"ContainerDied","Data":"27755c39299093d7bf79a29651e39c3c497060438cb6b14cf5c9154b62f4790c"}
Nov 03 08:05:07.756753 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:07.756730    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5"]
Nov 03 08:05:07.761072 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:07.761055    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5"
Nov 03 08:05:07.784882 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:07.784852    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5"]
Nov 03 08:05:07.861130 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:07.861101    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wtlt\" (UniqueName: \"kubernetes.io/projected/a18e750c-1f55-4bb7-8465-c5c03c406ee6-kube-api-access-7wtlt\") pod \"maas-api-revoke-sub-bdd7f375-spmj5\" (UID: \"a18e750c-1f55-4bb7-8465-c5c03c406ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5"
Nov 03 08:05:07.861261 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:07.861233    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18e750c-1f55-4bb7-8465-c5c03c406ee6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdd7f375-spmj5\" (UID: \"a18e750c-1f55-4bb7-8465-c5c03c406ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5"
Nov 03 08:05:07.861299 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:07.861290    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18e750c-1f55-4bb7-8465-c5c03c406ee6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdd7f375-spmj5\" (UID: \"a18e750c-1f55-4bb7-8465-c5c03c406ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5"
Nov 03 08:05:07.962444 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:07.962415    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wtlt\" (UniqueName: \"kubernetes.io/projected/a18e750c-1f55-4bb7-8465-c5c03c406ee6-kube-api-access-7wtlt\") pod \"maas-api-revoke-sub-bdd7f375-spmj5\" (UID: \"a18e750c-1f55-4bb7-8465-c5c03c406ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5"
Nov 03 08:05:07.962602 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:07.962564    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18e750c-1f55-4bb7-8465-c5c03c406ee6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdd7f375-spmj5\" (UID: \"a18e750c-1f55-4bb7-8465-c5c03c406ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5"
Nov 03 08:05:07.962665 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:07.962606    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18e750c-1f55-4bb7-8465-c5c03c406ee6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdd7f375-spmj5\" (UID: \"a18e750c-1f55-4bb7-8465-c5c03c406ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5"
Nov 03 08:05:07.963267 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:07.963245    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18e750c-1f55-4bb7-8465-c5c03c406ee6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdd7f375-spmj5\" (UID: \"a18e750c-1f55-4bb7-8465-c5c03c406ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5"
Nov 03 08:05:07.964923 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:07.964901    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18e750c-1f55-4bb7-8465-c5c03c406ee6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdd7f375-spmj5\" (UID: \"a18e750c-1f55-4bb7-8465-c5c03c406ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5"
Nov 03 08:05:07.969606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:07.969587    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wtlt\" (UniqueName: \"kubernetes.io/projected/a18e750c-1f55-4bb7-8465-c5c03c406ee6-kube-api-access-7wtlt\") pod \"maas-api-revoke-sub-bdd7f375-spmj5\" (UID: \"a18e750c-1f55-4bb7-8465-c5c03c406ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5"
Nov 03 08:05:08.075465 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.075440    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5"
Nov 03 08:05:08.356406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.356367    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5"]
Nov 03 08:05:08.365578 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:05:08.365538    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda18e750c_1f55_4bb7_8465_c5c03c406ee6.slice/crio-4ced7919448fbabc709dd4500ae7a026c792dfce7b1a9d7da6d02714d41866b8 WatchSource:0}: Error finding container 4ced7919448fbabc709dd4500ae7a026c792dfce7b1a9d7da6d02714d41866b8: Status 404 returned error can't find the container with id 4ced7919448fbabc709dd4500ae7a026c792dfce7b1a9d7da6d02714d41866b8
Nov 03 08:05:08.700071 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.700042    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5" event={"ID":"a18e750c-1f55-4bb7-8465-c5c03c406ee6","Type":"ContainerStarted","Data":"befd666de4c09a9bb15ddef0fd029234f7f2469b5980d9a013523c17043b16e5"}
Nov 03 08:05:08.700433 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.700095    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5" event={"ID":"a18e750c-1f55-4bb7-8465-c5c03c406ee6","Type":"ContainerStarted","Data":"4ced7919448fbabc709dd4500ae7a026c792dfce7b1a9d7da6d02714d41866b8"}
Nov 03 08:05:08.715750 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.715684    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5" podStartSLOduration=1.7156660129999999 podStartE2EDuration="1.715666013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:08.71312513 +0000 UTC m=+706.827589073" watchObservedRunningTime="2026-11-03 08:05:08.715666013 +0000 UTC m=+706.830129945"
Nov 03 08:05:08.880560 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.880527    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n"
Nov 03 08:05:08.922810 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.922727    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e33e4b-tm2n4"
Nov 03 08:05:08.945014 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.944991    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbef5506-hm49w"
Nov 03 08:05:08.975709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.975682    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3-maas-api-xks-ca\") pod \"9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3\" (UID: \"9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3\") "
Nov 03 08:05:08.975864 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.975724    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3-kube-api-access-8df5w\" (UniqueName: \"kubernetes.io/projected/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3-kube-api-access-8df5w\") pod \"9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3\" (UID: \"9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3\") "
Nov 03 08:05:08.975864 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.975747    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7db497ac-2392-4e2d-9b56-0398213ced6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db497ac-2392-4e2d-9b56-0398213ced6c-maas-api-openshift-ca\") pod \"7db497ac-2392-4e2d-9b56-0398213ced6c\" (UID: \"7db497ac-2392-4e2d-9b56-0398213ced6c\") "
Nov 03 08:05:08.975864 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.975777    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7db497ac-2392-4e2d-9b56-0398213ced6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db497ac-2392-4e2d-9b56-0398213ced6c-maas-api-xks-ca\") pod \"7db497ac-2392-4e2d-9b56-0398213ced6c\" (UID: \"7db497ac-2392-4e2d-9b56-0398213ced6c\") "
Nov 03 08:05:08.975864 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.975826    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00-kube-api-access-7r7xj\" (UniqueName: \"kubernetes.io/projected/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00-kube-api-access-7r7xj\") pod \"3fbc395a-05ae-4c54-a6a6-acdbc8e29f00\" (UID: \"3fbc395a-05ae-4c54-a6a6-acdbc8e29f00\") "
Nov 03 08:05:08.976087 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.975943    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00-maas-api-xks-ca\") pod \"3fbc395a-05ae-4c54-a6a6-acdbc8e29f00\" (UID: \"3fbc395a-05ae-4c54-a6a6-acdbc8e29f00\") "
Nov 03 08:05:08.976087 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.975971    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7db497ac-2392-4e2d-9b56-0398213ced6c-kube-api-access-642kc\" (UniqueName: \"kubernetes.io/projected/7db497ac-2392-4e2d-9b56-0398213ced6c-kube-api-access-642kc\") pod \"7db497ac-2392-4e2d-9b56-0398213ced6c\" (UID: \"7db497ac-2392-4e2d-9b56-0398213ced6c\") "
Nov 03 08:05:08.976087 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.976005    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00-maas-api-openshift-ca\") pod \"3fbc395a-05ae-4c54-a6a6-acdbc8e29f00\" (UID: \"3fbc395a-05ae-4c54-a6a6-acdbc8e29f00\") "
Nov 03 08:05:08.976087 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.976037    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3-maas-api-openshift-ca\") pod \"9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3\" (UID: \"9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3\") "
Nov 03 08:05:08.977062 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.976843    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00-maas-api-openshift-ca" pod "3fbc395a-05ae-4c54-a6a6-acdbc8e29f00" (UID: "3fbc395a-05ae-4c54-a6a6-acdbc8e29f00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:05:08.977165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.977134    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7db497ac-2392-4e2d-9b56-0398213ced6c-maas-api-openshift-ca" pod "7db497ac-2392-4e2d-9b56-0398213ced6c" (UID: "7db497ac-2392-4e2d-9b56-0398213ced6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:05:08.977999 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.977971    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3-maas-api-openshift-ca" pod "9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3" (UID: "9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:05:08.978925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.978900    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7db497ac-2392-4e2d-9b56-0398213ced6c-maas-api-xks-ca" pod "7db497ac-2392-4e2d-9b56-0398213ced6c" (UID: "7db497ac-2392-4e2d-9b56-0398213ced6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:05:08.978925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.978915    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3-maas-api-xks-ca" pod "9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3" (UID: "9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:05:08.979092 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.979016    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3-kube-api-access-8df5w" pod "9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3" (UID: "9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3"). InnerVolumeSpecName "kube-api-access-8df5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:05:08.979092 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.979074    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00-kube-api-access-7r7xj" pod "3fbc395a-05ae-4c54-a6a6-acdbc8e29f00" (UID: "3fbc395a-05ae-4c54-a6a6-acdbc8e29f00"). InnerVolumeSpecName "kube-api-access-7r7xj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:05:08.979280 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.979243    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00-maas-api-xks-ca" pod "3fbc395a-05ae-4c54-a6a6-acdbc8e29f00" (UID: "3fbc395a-05ae-4c54-a6a6-acdbc8e29f00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:05:08.980217 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:08.980192    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7db497ac-2392-4e2d-9b56-0398213ced6c-kube-api-access-642kc" pod "7db497ac-2392-4e2d-9b56-0398213ced6c" (UID: "7db497ac-2392-4e2d-9b56-0398213ced6c"). InnerVolumeSpecName "kube-api-access-642kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:05:09.077360 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:09.077336    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:09.077360 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:09.077361    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8df5w\" (UniqueName: \"kubernetes.io/projected/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3-kube-api-access-8df5w\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:09.077503 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:09.077372    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7db497ac-2392-4e2d-9b56-0398213ced6c-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:09.077503 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:09.077382    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7db497ac-2392-4e2d-9b56-0398213ced6c-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:09.077503 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:09.077391    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r7xj\" (UniqueName: \"kubernetes.io/projected/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00-kube-api-access-7r7xj\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:09.077503 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:09.077399    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:09.077503 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:09.077407    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-642kc\" (UniqueName: \"kubernetes.io/projected/7db497ac-2392-4e2d-9b56-0398213ced6c-kube-api-access-642kc\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:09.077503 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:09.077416    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:09.077503 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:09.077424    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:09.708163 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:09.708137    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbef5506-hm49w"
Nov 03 08:05:09.708583 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:09.708138    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbef5506-hm49w" event={"ID":"7db497ac-2392-4e2d-9b56-0398213ced6c","Type":"ContainerDied","Data":"400f8bb8384b73101196680842e10125a2120ee946fb70ef1fb05ac3044a898a"}
Nov 03 08:05:09.708583 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:09.708270    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="400f8bb8384b73101196680842e10125a2120ee946fb70ef1fb05ac3044a898a"
Nov 03 08:05:09.709435 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:09.709414    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e33e4b-tm2n4"
Nov 03 08:05:09.709575 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:09.709470    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4e33e4b-tm2n4" event={"ID":"9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3","Type":"ContainerDied","Data":"db8c45acf6c7227ac8bb8a43b3eb8cdd6e82916aeecc6519cd6dbd3322cfd4f6"}
Nov 03 08:05:09.709575 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:09.709512    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db8c45acf6c7227ac8bb8a43b3eb8cdd6e82916aeecc6519cd6dbd3322cfd4f6"
Nov 03 08:05:09.711103 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:09.711077    5070 generic.go:356] "Generic (PLEG): container finished" podID="a18e750c-1f55-4bb7-8465-c5c03c406ee6" containerID="befd666de4c09a9bb15ddef0fd029234f7f2469b5980d9a013523c17043b16e5" exitCode=0
Nov 03 08:05:09.711220 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:09.711168    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5" event={"ID":"a18e750c-1f55-4bb7-8465-c5c03c406ee6","Type":"ContainerDied","Data":"befd666de4c09a9bb15ddef0fd029234f7f2469b5980d9a013523c17043b16e5"}
Nov 03 08:05:09.712557 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:09.712535    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n" event={"ID":"3fbc395a-05ae-4c54-a6a6-acdbc8e29f00","Type":"ContainerDied","Data":"acc17bdb60193777110fc13abac3d391330e9e5468743808a496dff3912c15b7"}
Nov 03 08:05:09.712659 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:09.712571    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acc17bdb60193777110fc13abac3d391330e9e5468743808a496dff3912c15b7"
Nov 03 08:05:09.712659 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:09.712539    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n"
Nov 03 08:05:10.909287 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:10.909267    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5"
Nov 03 08:05:10.997961 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:10.997893    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a18e750c-1f55-4bb7-8465-c5c03c406ee6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18e750c-1f55-4bb7-8465-c5c03c406ee6-maas-api-xks-ca\") pod \"a18e750c-1f55-4bb7-8465-c5c03c406ee6\" (UID: \"a18e750c-1f55-4bb7-8465-c5c03c406ee6\") "
Nov 03 08:05:10.998094 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:10.997998    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a18e750c-1f55-4bb7-8465-c5c03c406ee6-kube-api-access-7wtlt\" (UniqueName: \"kubernetes.io/projected/a18e750c-1f55-4bb7-8465-c5c03c406ee6-kube-api-access-7wtlt\") pod \"a18e750c-1f55-4bb7-8465-c5c03c406ee6\" (UID: \"a18e750c-1f55-4bb7-8465-c5c03c406ee6\") "
Nov 03 08:05:10.998094 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:10.998021    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a18e750c-1f55-4bb7-8465-c5c03c406ee6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18e750c-1f55-4bb7-8465-c5c03c406ee6-maas-api-openshift-ca\") pod \"a18e750c-1f55-4bb7-8465-c5c03c406ee6\" (UID: \"a18e750c-1f55-4bb7-8465-c5c03c406ee6\") "
Nov 03 08:05:10.998472 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:10.998442    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a18e750c-1f55-4bb7-8465-c5c03c406ee6-maas-api-openshift-ca" pod "a18e750c-1f55-4bb7-8465-c5c03c406ee6" (UID: "a18e750c-1f55-4bb7-8465-c5c03c406ee6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:05:10.999892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:10.999874    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a18e750c-1f55-4bb7-8465-c5c03c406ee6-maas-api-xks-ca" pod "a18e750c-1f55-4bb7-8465-c5c03c406ee6" (UID: "a18e750c-1f55-4bb7-8465-c5c03c406ee6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:05:11.000116 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:11.000093    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a18e750c-1f55-4bb7-8465-c5c03c406ee6-kube-api-access-7wtlt" pod "a18e750c-1f55-4bb7-8465-c5c03c406ee6" (UID: "a18e750c-1f55-4bb7-8465-c5c03c406ee6"). InnerVolumeSpecName "kube-api-access-7wtlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:05:11.099689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:11.099655    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a18e750c-1f55-4bb7-8465-c5c03c406ee6-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:11.099689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:11.099685    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wtlt\" (UniqueName: \"kubernetes.io/projected/a18e750c-1f55-4bb7-8465-c5c03c406ee6-kube-api-access-7wtlt\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:11.099865 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:11.099700    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a18e750c-1f55-4bb7-8465-c5c03c406ee6-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:11.727008 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:11.726976    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5"
Nov 03 08:05:11.727172 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:11.727005    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5" event={"ID":"a18e750c-1f55-4bb7-8465-c5c03c406ee6","Type":"ContainerDied","Data":"4ced7919448fbabc709dd4500ae7a026c792dfce7b1a9d7da6d02714d41866b8"}
Nov 03 08:05:11.727172 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:11.727054    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ced7919448fbabc709dd4500ae7a026c792dfce7b1a9d7da6d02714d41866b8"
Nov 03 08:05:15.337791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.336209    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"]
Nov 03 08:05:15.337791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.337089    5070 cpu_manager.go:414] "containerMap: removing container" podUID="9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3" containerName="revoke-keys"
Nov 03 08:05:15.337791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.337108    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3" containerName="revoke-keys"
Nov 03 08:05:15.337791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.337159    5070 cpu_manager.go:414] "containerMap: removing container" podUID="7db497ac-2392-4e2d-9b56-0398213ced6c" containerName="revoke-keys"
Nov 03 08:05:15.337791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.337167    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7db497ac-2392-4e2d-9b56-0398213ced6c" containerName="revoke-keys"
Nov 03 08:05:15.337791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.337186    5070 cpu_manager.go:414] "containerMap: removing container" podUID="a18e750c-1f55-4bb7-8465-c5c03c406ee6" containerName="revoke-keys"
Nov 03 08:05:15.337791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.337194    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a18e750c-1f55-4bb7-8465-c5c03c406ee6" containerName="revoke-keys"
Nov 03 08:05:15.337791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.337212    5070 cpu_manager.go:414] "containerMap: removing container" podUID="3fbc395a-05ae-4c54-a6a6-acdbc8e29f00" containerName="revoke-keys"
Nov 03 08:05:15.337791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.337221    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fbc395a-05ae-4c54-a6a6-acdbc8e29f00" containerName="revoke-keys"
Nov 03 08:05:15.337791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.337340    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3" containerName="revoke-keys"
Nov 03 08:05:15.337791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.337352    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="7db497ac-2392-4e2d-9b56-0398213ced6c" containerName="revoke-keys"
Nov 03 08:05:15.337791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.337363    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="a18e750c-1f55-4bb7-8465-c5c03c406ee6" containerName="revoke-keys"
Nov 03 08:05:15.337791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.337374    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fbc395a-05ae-4c54-a6a6-acdbc8e29f00" containerName="revoke-keys"
Nov 03 08:05:15.530783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.521721    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"]
Nov 03 08:05:15.555905 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.523900    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:15.763786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.733373    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:15.770857 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.770829    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181a2d26-3317-450e-97be-c0082d47a27a-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:15.771083 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.771060    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:15.771237 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.771218    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:15.774867 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.774842    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181a2d26-3317-450e-97be-c0082d47a27a-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:15.775051 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.775031    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:15.775196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.775177    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:15.775333 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.775316    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjlps\" (UniqueName: \"kubernetes.io/projected/181a2d26-3317-450e-97be-c0082d47a27a-kube-api-access-cjlps\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:15.775559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.775538    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181a2d26-3317-450e-97be-c0082d47a27a-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:15.918781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.916853    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181a2d26-3317-450e-97be-c0082d47a27a-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:15.918781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.916966    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:15.918781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.916998    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181a2d26-3317-450e-97be-c0082d47a27a-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:15.918781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.917032    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:15.918781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.917075    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:15.918781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.917125    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181a2d26-3317-450e-97be-c0082d47a27a-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:15.918781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.917181    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:15.918781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.917224    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:15.918781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:15.917268    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjlps\" (UniqueName: \"kubernetes.io/projected/181a2d26-3317-450e-97be-c0082d47a27a-kube-api-access-cjlps\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:16.006783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:16.002229    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:16.007545 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:16.007516    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:16.040836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:16.020069    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:16.040836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:16.021632    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:16.101794 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:16.097881    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181a2d26-3317-450e-97be-c0082d47a27a-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:16.101794 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:16.098177    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:16.336832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:16.332574    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181a2d26-3317-450e-97be-c0082d47a27a-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:16.339923 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:16.339886    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjlps\" (UniqueName: \"kubernetes.io/projected/181a2d26-3317-450e-97be-c0082d47a27a-kube-api-access-cjlps\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:16.340975 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:16.340949    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181a2d26-3317-450e-97be-c0082d47a27a-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:16.436215 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:16.436174    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:16.484193 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:16.484150    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:16.509942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:16.509900    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:16.573782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:16.550305    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:16.775782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:16.752714    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:16.775782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:16.753067    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:05:16.830782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:16.796291    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:16.830782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:16.823488    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:17.266490 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:17.266459    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx"]
Nov 03 08:05:17.287381 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:17.287341    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx"]
Nov 03 08:05:17.287524 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:17.287474    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx"
Nov 03 08:05:17.293130 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:17.292700    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:05:17.293130 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:17.292794    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:05:17.293130 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:17.292872    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:05:17.299640 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:17.299619    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:05:17.459976 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:17.459879    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmx62\" (UniqueName: \"kubernetes.io/projected/4397fae5-589b-4a11-9bfc-fefce07a5ab6-kube-api-access-vmx62\") pod \"maas-api-revoke-sub-75f83b59-l54bx\" (UID: \"4397fae5-589b-4a11-9bfc-fefce07a5ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx"
Nov 03 08:05:17.460619 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:17.460581    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4397fae5-589b-4a11-9bfc-fefce07a5ab6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75f83b59-l54bx\" (UID: \"4397fae5-589b-4a11-9bfc-fefce07a5ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx"
Nov 03 08:05:17.460915 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:17.460887    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4397fae5-589b-4a11-9bfc-fefce07a5ab6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75f83b59-l54bx\" (UID: \"4397fae5-589b-4a11-9bfc-fefce07a5ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx"
Nov 03 08:05:17.567571 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:17.567533    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmx62\" (UniqueName: \"kubernetes.io/projected/4397fae5-589b-4a11-9bfc-fefce07a5ab6-kube-api-access-vmx62\") pod \"maas-api-revoke-sub-75f83b59-l54bx\" (UID: \"4397fae5-589b-4a11-9bfc-fefce07a5ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx"
Nov 03 08:05:17.568229 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:17.568184    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4397fae5-589b-4a11-9bfc-fefce07a5ab6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75f83b59-l54bx\" (UID: \"4397fae5-589b-4a11-9bfc-fefce07a5ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx"
Nov 03 08:05:17.568998 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:17.568964    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4397fae5-589b-4a11-9bfc-fefce07a5ab6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75f83b59-l54bx\" (UID: \"4397fae5-589b-4a11-9bfc-fefce07a5ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx"
Nov 03 08:05:17.591878 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:17.569833    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4397fae5-589b-4a11-9bfc-fefce07a5ab6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75f83b59-l54bx\" (UID: \"4397fae5-589b-4a11-9bfc-fefce07a5ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx"
Nov 03 08:05:17.592787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:17.592716    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmx62\" (UniqueName: \"kubernetes.io/projected/4397fae5-589b-4a11-9bfc-fefce07a5ab6-kube-api-access-vmx62\") pod \"maas-api-revoke-sub-75f83b59-l54bx\" (UID: \"4397fae5-589b-4a11-9bfc-fefce07a5ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx"
Nov 03 08:05:17.606958 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:17.606928    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4397fae5-589b-4a11-9bfc-fefce07a5ab6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75f83b59-l54bx\" (UID: \"4397fae5-589b-4a11-9bfc-fefce07a5ab6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx"
Nov 03 08:05:17.608879 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:17.608837    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:05:17.608967 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:17.608931    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:05:17.609042 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:17.608980    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:05:17.626873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:17.626619    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx"
Nov 03 08:05:17.644339 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:17.642402    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"]
Nov 03 08:05:18.149009 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:05:18.148962    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4397fae5_589b_4a11_9bfc_fefce07a5ab6.slice/crio-17a5f7e026f40c42d29f6f0dddcd0cd35fd723071e2725bb8d3a9ae31201a279 WatchSource:0}: Error finding container 17a5f7e026f40c42d29f6f0dddcd0cd35fd723071e2725bb8d3a9ae31201a279: Status 404 returned error can't find the container with id 17a5f7e026f40c42d29f6f0dddcd0cd35fd723071e2725bb8d3a9ae31201a279
Nov 03 08:05:18.149813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:18.149792    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx"]
Nov 03 08:05:18.309195 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:18.308125    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r" event={"ID":"181a2d26-3317-450e-97be-c0082d47a27a","Type":"ContainerStarted","Data":"c96a23ab691a87d470784c9aa049512543d087810e8fc7695666836b49509f61"}
Nov 03 08:05:18.309195 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:18.308193    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r" event={"ID":"181a2d26-3317-450e-97be-c0082d47a27a","Type":"ContainerStarted","Data":"f77b03bc766f4214fe8c69fc350806398936e1d1b53358fc8863bca4d8709370"}
Nov 03 08:05:18.312373 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:18.312316    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx" event={"ID":"4397fae5-589b-4a11-9bfc-fefce07a5ab6","Type":"ContainerStarted","Data":"17a5f7e026f40c42d29f6f0dddcd0cd35fd723071e2725bb8d3a9ae31201a279"}
Nov 03 08:05:18.353624 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:18.353571    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r" podStartSLOduration=3.353551711 podStartE2EDuration="3.353551711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:18.346432591 +0000 UTC m=+716.460896526" watchObservedRunningTime="2026-11-03 08:05:18.353551711 +0000 UTC m=+716.468015644"
Nov 03 08:05:18.444951 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:18.444835    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:19.322185 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:19.322123    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx" event={"ID":"4397fae5-589b-4a11-9bfc-fefce07a5ab6","Type":"ContainerStarted","Data":"e4f41305f818c2ad9cc7813569a1073171b9f50ff470a454b6af4857c1aa7b0e"}
Nov 03 08:05:19.338481 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:19.338438    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx" podStartSLOduration=3.338425597 podStartE2EDuration="3.338425597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:19.334884069 +0000 UTC m=+717.449348000" watchObservedRunningTime="2026-11-03 08:05:19.338425597 +0000 UTC m=+717.452889527"
Nov 03 08:05:19.443302 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:19.443269    5070 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 08:05:19.443466 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:19.443328    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r" podUID="181a2d26-3317-450e-97be-c0082d47a27a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 08:05:19.700845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:19.700563    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95"]
Nov 03 08:05:19.705413 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:19.705384    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95"
Nov 03 08:05:19.705553 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:19.705462    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95"]
Nov 03 08:05:19.843363 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:19.843325    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e65e7fd-jmq95\" (UID: \"106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95"
Nov 03 08:05:19.843621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:19.843466    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79jjp\" (UniqueName: \"kubernetes.io/projected/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d-kube-api-access-79jjp\") pod \"maas-api-revoke-sub-5e65e7fd-jmq95\" (UID: \"106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95"
Nov 03 08:05:19.843621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:19.843526    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e65e7fd-jmq95\" (UID: \"106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95"
Nov 03 08:05:19.908892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:19.908861    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l"]
Nov 03 08:05:19.913286 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:19.913269    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l"
Nov 03 08:05:19.921129 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:19.921100    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l"]
Nov 03 08:05:19.946943 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:19.946907    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e65e7fd-jmq95\" (UID: \"106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95"
Nov 03 08:05:19.947096 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:19.947043    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79jjp\" (UniqueName: \"kubernetes.io/projected/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d-kube-api-access-79jjp\") pod \"maas-api-revoke-sub-5e65e7fd-jmq95\" (UID: \"106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95"
Nov 03 08:05:19.947143 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:19.947098    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e65e7fd-jmq95\" (UID: \"106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95"
Nov 03 08:05:19.948478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:19.948449    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e65e7fd-jmq95\" (UID: \"106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95"
Nov 03 08:05:19.950126 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:19.950095    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e65e7fd-jmq95\" (UID: \"106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95"
Nov 03 08:05:19.955745 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:19.955686    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79jjp\" (UniqueName: \"kubernetes.io/projected/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d-kube-api-access-79jjp\") pod \"maas-api-revoke-sub-5e65e7fd-jmq95\" (UID: \"106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95"
Nov 03 08:05:20.020631 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:20.020598    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95"
Nov 03 08:05:20.048318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:20.048290    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fd1cadf-e4be-4d9c-be54-b67c399b0556-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51797970-6m56l\" (UID: \"6fd1cadf-e4be-4d9c-be54-b67c399b0556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l"
Nov 03 08:05:20.048494 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:20.048472    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fd1cadf-e4be-4d9c-be54-b67c399b0556-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51797970-6m56l\" (UID: \"6fd1cadf-e4be-4d9c-be54-b67c399b0556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l"
Nov 03 08:05:20.048578 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:20.048564    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vmpq\" (UniqueName: \"kubernetes.io/projected/6fd1cadf-e4be-4d9c-be54-b67c399b0556-kube-api-access-5vmpq\") pod \"maas-api-revoke-sub-51797970-6m56l\" (UID: \"6fd1cadf-e4be-4d9c-be54-b67c399b0556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l"
Nov 03 08:05:20.153784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:20.150553    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fd1cadf-e4be-4d9c-be54-b67c399b0556-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51797970-6m56l\" (UID: \"6fd1cadf-e4be-4d9c-be54-b67c399b0556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l"
Nov 03 08:05:20.153784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:20.150604    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmpq\" (UniqueName: \"kubernetes.io/projected/6fd1cadf-e4be-4d9c-be54-b67c399b0556-kube-api-access-5vmpq\") pod \"maas-api-revoke-sub-51797970-6m56l\" (UID: \"6fd1cadf-e4be-4d9c-be54-b67c399b0556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l"
Nov 03 08:05:20.153784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:20.150682    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fd1cadf-e4be-4d9c-be54-b67c399b0556-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51797970-6m56l\" (UID: \"6fd1cadf-e4be-4d9c-be54-b67c399b0556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l"
Nov 03 08:05:20.153784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:20.151277    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fd1cadf-e4be-4d9c-be54-b67c399b0556-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51797970-6m56l\" (UID: \"6fd1cadf-e4be-4d9c-be54-b67c399b0556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l"
Nov 03 08:05:20.153784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:20.153355    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fd1cadf-e4be-4d9c-be54-b67c399b0556-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51797970-6m56l\" (UID: \"6fd1cadf-e4be-4d9c-be54-b67c399b0556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l"
Nov 03 08:05:20.182685 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:20.182348    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vmpq\" (UniqueName: \"kubernetes.io/projected/6fd1cadf-e4be-4d9c-be54-b67c399b0556-kube-api-access-5vmpq\") pod \"maas-api-revoke-sub-51797970-6m56l\" (UID: \"6fd1cadf-e4be-4d9c-be54-b67c399b0556\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l"
Nov 03 08:05:20.235520 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:20.235460    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l"
Nov 03 08:05:20.327090 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:20.326114    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95"]
Nov 03 08:05:20.338126 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:20.338102    5070 generic.go:356] "Generic (PLEG): container finished" podID="4397fae5-589b-4a11-9bfc-fefce07a5ab6" containerID="e4f41305f818c2ad9cc7813569a1073171b9f50ff470a454b6af4857c1aa7b0e" exitCode=0
Nov 03 08:05:20.338340 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:20.338321    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx" event={"ID":"4397fae5-589b-4a11-9bfc-fefce07a5ab6","Type":"ContainerDied","Data":"e4f41305f818c2ad9cc7813569a1073171b9f50ff470a454b6af4857c1aa7b0e"}
Nov 03 08:05:20.442611 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:20.442584    5070 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 08:05:20.442737 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:20.442643    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r" podUID="181a2d26-3317-450e-97be-c0082d47a27a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 08:05:20.536621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:20.536587    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l"]
Nov 03 08:05:20.537140 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:05:20.537115    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd1cadf_e4be_4d9c_be54_b67c399b0556.slice/crio-67fc1e379a1ecfffb59820885cedc68e10a177b64899ed954aa84df63f53bccd WatchSource:0}: Error finding container 67fc1e379a1ecfffb59820885cedc68e10a177b64899ed954aa84df63f53bccd: Status 404 returned error can't find the container with id 67fc1e379a1ecfffb59820885cedc68e10a177b64899ed954aa84df63f53bccd
Nov 03 08:05:21.347307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.347272    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l" event={"ID":"6fd1cadf-e4be-4d9c-be54-b67c399b0556","Type":"ContainerStarted","Data":"5e413b0b49e17818550abb8d1cd332516efbe398bce9e0ad5dbb906985ebb2e4"}
Nov 03 08:05:21.347749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.347329    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l" event={"ID":"6fd1cadf-e4be-4d9c-be54-b67c399b0556","Type":"ContainerStarted","Data":"67fc1e379a1ecfffb59820885cedc68e10a177b64899ed954aa84df63f53bccd"}
Nov 03 08:05:21.349019 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.348985    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95" event={"ID":"106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d","Type":"ContainerStarted","Data":"dd9b7587b2e6dae86b36e6e29a5fc1bd8dd8a3bfd10c68cb0dd9de3899fb295a"}
Nov 03 08:05:21.349139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.349037    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95" event={"ID":"106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d","Type":"ContainerStarted","Data":"e98fa764d99bff105a700c59310e03ae91ac94826c04ef1a669a551362b363d2"}
Nov 03 08:05:21.388545 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.388489    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l" podStartSLOduration=2.388476412 podStartE2EDuration="2.388476412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:21.384979134 +0000 UTC m=+719.499443066" watchObservedRunningTime="2026-11-03 08:05:21.388476412 +0000 UTC m=+719.502940342"
Nov 03 08:05:21.404586 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.404524    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95" podStartSLOduration=2.404504866 podStartE2EDuration="2.404504866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:21.399431143 +0000 UTC m=+719.513895074" watchObservedRunningTime="2026-11-03 08:05:21.404504866 +0000 UTC m=+719.518968863"
Nov 03 08:05:21.443589 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.443550    5070 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 08:05:21.444943 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.443604    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r" podUID="181a2d26-3317-450e-97be-c0082d47a27a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 08:05:21.447936 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.447913    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"]
Nov 03 08:05:21.454647 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.453383    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.496608 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.496518    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"]
Nov 03 08:05:21.595817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.595789    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx"
Nov 03 08:05:21.598785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.598712    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33edac1-844e-4caf-bb59-d63291659555-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.598785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.598775    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.598955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.598886    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.598955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.598911    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6qgv\" (UniqueName: \"kubernetes.io/projected/e33edac1-844e-4caf-bb59-d63291659555-kube-api-access-r6qgv\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.598955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.598940    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.599168 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.599147    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.599229 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.599189    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.599281 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.599260    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33edac1-844e-4caf-bb59-d63291659555-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.599336 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.599288    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33edac1-844e-4caf-bb59-d63291659555-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.704021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.702211    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4397fae5-589b-4a11-9bfc-fefce07a5ab6-kube-api-access-vmx62\" (UniqueName: \"kubernetes.io/projected/4397fae5-589b-4a11-9bfc-fefce07a5ab6-kube-api-access-vmx62\") pod \"4397fae5-589b-4a11-9bfc-fefce07a5ab6\" (UID: \"4397fae5-589b-4a11-9bfc-fefce07a5ab6\") "
Nov 03 08:05:21.704021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.702260    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4397fae5-589b-4a11-9bfc-fefce07a5ab6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4397fae5-589b-4a11-9bfc-fefce07a5ab6-maas-api-xks-ca\") pod \"4397fae5-589b-4a11-9bfc-fefce07a5ab6\" (UID: \"4397fae5-589b-4a11-9bfc-fefce07a5ab6\") "
Nov 03 08:05:21.704021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.702364    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4397fae5-589b-4a11-9bfc-fefce07a5ab6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4397fae5-589b-4a11-9bfc-fefce07a5ab6-maas-api-openshift-ca\") pod \"4397fae5-589b-4a11-9bfc-fefce07a5ab6\" (UID: \"4397fae5-589b-4a11-9bfc-fefce07a5ab6\") "
Nov 03 08:05:21.704021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.702698    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.704021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.702733    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.704021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.702753    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33edac1-844e-4caf-bb59-d63291659555-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.704021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.702785    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33edac1-844e-4caf-bb59-d63291659555-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.704021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.702895    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33edac1-844e-4caf-bb59-d63291659555-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.704021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.702928    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.704021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.702992    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.704021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.703050    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6qgv\" (UniqueName: \"kubernetes.io/projected/e33edac1-844e-4caf-bb59-d63291659555-kube-api-access-r6qgv\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.704021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.703065    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.704021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.703101    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.704021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.703474    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.704021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.703574    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.704021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.703831    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4397fae5-589b-4a11-9bfc-fefce07a5ab6-maas-api-openshift-ca" pod "4397fae5-589b-4a11-9bfc-fefce07a5ab6" (UID: "4397fae5-589b-4a11-9bfc-fefce07a5ab6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:05:21.704021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.703968    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33edac1-844e-4caf-bb59-d63291659555-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.704021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.704004    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.705924 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.705898    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4397fae5-589b-4a11-9bfc-fefce07a5ab6-maas-api-xks-ca" pod "4397fae5-589b-4a11-9bfc-fefce07a5ab6" (UID: "4397fae5-589b-4a11-9bfc-fefce07a5ab6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:05:21.706014 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.705912    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4397fae5-589b-4a11-9bfc-fefce07a5ab6-kube-api-access-vmx62" pod "4397fae5-589b-4a11-9bfc-fefce07a5ab6" (UID: "4397fae5-589b-4a11-9bfc-fefce07a5ab6"). InnerVolumeSpecName "kube-api-access-vmx62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:05:21.706014 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.705956    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.709174 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.708412    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33edac1-844e-4caf-bb59-d63291659555-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.709783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.709745    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33edac1-844e-4caf-bb59-d63291659555-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.710033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.710009    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6qgv\" (UniqueName: \"kubernetes.io/projected/e33edac1-844e-4caf-bb59-d63291659555-kube-api-access-r6qgv\") pod \"maas-default-gateway-openshift-default-76f4876667-l6bvc\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:21.804209 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.804175    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmx62\" (UniqueName: \"kubernetes.io/projected/4397fae5-589b-4a11-9bfc-fefce07a5ab6-kube-api-access-vmx62\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:21.804209 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.804204    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4397fae5-589b-4a11-9bfc-fefce07a5ab6-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:21.804209 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.804214    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4397fae5-589b-4a11-9bfc-fefce07a5ab6-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:21.910751 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:21.910717    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:22.052557 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:05:22.052521    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:05:22.225155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:22.225122    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"]
Nov 03 08:05:22.225266 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:05:22.225244    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode33edac1_844e_4caf_bb59_d63291659555.slice/crio-d0ef283949ee7c95f36a8ad159811c81e199eb0cea967d6a262edf410e12edbc WatchSource:0}: Error finding container d0ef283949ee7c95f36a8ad159811c81e199eb0cea967d6a262edf410e12edbc: Status 404 returned error can't find the container with id d0ef283949ee7c95f36a8ad159811c81e199eb0cea967d6a262edf410e12edbc
Nov 03 08:05:22.227612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:22.227574    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:05:22.227734 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:22.227648    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:05:22.227734 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:22.227692    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:05:22.379780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:22.377724    5070 generic.go:356] "Generic (PLEG): container finished" podID="106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d" containerID="dd9b7587b2e6dae86b36e6e29a5fc1bd8dd8a3bfd10c68cb0dd9de3899fb295a" exitCode=0
Nov 03 08:05:22.379780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:22.377863    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95" event={"ID":"106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d","Type":"ContainerDied","Data":"dd9b7587b2e6dae86b36e6e29a5fc1bd8dd8a3bfd10c68cb0dd9de3899fb295a"}
Nov 03 08:05:22.388455 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:22.387959    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc" event={"ID":"e33edac1-844e-4caf-bb59-d63291659555","Type":"ContainerStarted","Data":"7d1c88516aa76e6bfff3cf5706a8fcfa0cd75b58781094bcad099e5871bedc1b"}
Nov 03 08:05:22.388455 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:22.388014    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc" event={"ID":"e33edac1-844e-4caf-bb59-d63291659555","Type":"ContainerStarted","Data":"d0ef283949ee7c95f36a8ad159811c81e199eb0cea967d6a262edf410e12edbc"}
Nov 03 08:05:22.389691 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:22.389664    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx" event={"ID":"4397fae5-589b-4a11-9bfc-fefce07a5ab6","Type":"ContainerDied","Data":"17a5f7e026f40c42d29f6f0dddcd0cd35fd723071e2725bb8d3a9ae31201a279"}
Nov 03 08:05:22.389861 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:22.389752    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17a5f7e026f40c42d29f6f0dddcd0cd35fd723071e2725bb8d3a9ae31201a279"
Nov 03 08:05:22.389861 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:22.389670    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx"
Nov 03 08:05:22.392465 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:22.392441    5070 generic.go:356] "Generic (PLEG): container finished" podID="6fd1cadf-e4be-4d9c-be54-b67c399b0556" containerID="5e413b0b49e17818550abb8d1cd332516efbe398bce9e0ad5dbb906985ebb2e4" exitCode=0
Nov 03 08:05:22.392581 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:22.392499    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l" event={"ID":"6fd1cadf-e4be-4d9c-be54-b67c399b0556","Type":"ContainerDied","Data":"5e413b0b49e17818550abb8d1cd332516efbe398bce9e0ad5dbb906985ebb2e4"}
Nov 03 08:05:22.439935 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:22.439890    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc" podStartSLOduration=1.439876678 podStartE2EDuration="1.439876678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:22.435131537 +0000 UTC m=+720.549595469" watchObservedRunningTime="2026-11-03 08:05:22.439876678 +0000 UTC m=+720.554340609"
Nov 03 08:05:22.443011 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:22.442987    5070 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 08:05:22.443125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:22.443036    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r" podUID="181a2d26-3317-450e-97be-c0082d47a27a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 08:05:22.911460 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:22.911417    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:23.442552 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:23.442503    5070 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 08:05:23.444209 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:23.443956    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r" podUID="181a2d26-3317-450e-97be-c0082d47a27a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 08:05:23.848163 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:23.848132    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l"
Nov 03 08:05:23.850501 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:23.850477    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95"
Nov 03 08:05:23.920497 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:23.920461    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-l6bvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 08:05:23.920820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:23.920751    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc" podUID="e33edac1-844e-4caf-bb59-d63291659555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 08:05:23.939302 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:23.939243    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fd1cadf-e4be-4d9c-be54-b67c399b0556-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fd1cadf-e4be-4d9c-be54-b67c399b0556-maas-api-xks-ca\") pod \"6fd1cadf-e4be-4d9c-be54-b67c399b0556\" (UID: \"6fd1cadf-e4be-4d9c-be54-b67c399b0556\") "
Nov 03 08:05:23.939434 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:23.939404    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d-maas-api-openshift-ca\") pod \"106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d\" (UID: \"106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d\") "
Nov 03 08:05:23.939520 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:23.939501    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fd1cadf-e4be-4d9c-be54-b67c399b0556-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fd1cadf-e4be-4d9c-be54-b67c399b0556-maas-api-openshift-ca\") pod \"6fd1cadf-e4be-4d9c-be54-b67c399b0556\" (UID: \"6fd1cadf-e4be-4d9c-be54-b67c399b0556\") "
Nov 03 08:05:23.939867 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:23.939841    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d-maas-api-xks-ca\") pod \"106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d\" (UID: \"106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d\") "
Nov 03 08:05:23.940059 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:23.940037    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d-maas-api-openshift-ca" pod "106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d" (UID: "106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:05:23.940408 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:23.940384    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:23.940569 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:23.940549    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fd1cadf-e4be-4d9c-be54-b67c399b0556-maas-api-openshift-ca" pod "6fd1cadf-e4be-4d9c-be54-b67c399b0556" (UID: "6fd1cadf-e4be-4d9c-be54-b67c399b0556"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:05:23.942552 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:23.942524    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fd1cadf-e4be-4d9c-be54-b67c399b0556-maas-api-xks-ca" pod "6fd1cadf-e4be-4d9c-be54-b67c399b0556" (UID: "6fd1cadf-e4be-4d9c-be54-b67c399b0556"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:05:23.942653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:23.942609    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d-maas-api-xks-ca" pod "106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d" (UID: "106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:05:24.041322 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:24.041291    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fd1cadf-e4be-4d9c-be54-b67c399b0556-kube-api-access-5vmpq\" (UniqueName: \"kubernetes.io/projected/6fd1cadf-e4be-4d9c-be54-b67c399b0556-kube-api-access-5vmpq\") pod \"6fd1cadf-e4be-4d9c-be54-b67c399b0556\" (UID: \"6fd1cadf-e4be-4d9c-be54-b67c399b0556\") "
Nov 03 08:05:24.041515 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:24.041344    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d-kube-api-access-79jjp\" (UniqueName: \"kubernetes.io/projected/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d-kube-api-access-79jjp\") pod \"106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d\" (UID: \"106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d\") "
Nov 03 08:05:24.041667 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:24.041652    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fd1cadf-e4be-4d9c-be54-b67c399b0556-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:24.041667 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:24.041668    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:24.041828 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:24.041680    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fd1cadf-e4be-4d9c-be54-b67c399b0556-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:24.044044 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:24.043999    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d-kube-api-access-79jjp" pod "106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d" (UID: "106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d"). InnerVolumeSpecName "kube-api-access-79jjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:05:24.044307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:24.044289    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd1cadf-e4be-4d9c-be54-b67c399b0556-kube-api-access-5vmpq" pod "6fd1cadf-e4be-4d9c-be54-b67c399b0556" (UID: "6fd1cadf-e4be-4d9c-be54-b67c399b0556"). InnerVolumeSpecName "kube-api-access-5vmpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:05:24.143856 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:24.143822    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vmpq\" (UniqueName: \"kubernetes.io/projected/6fd1cadf-e4be-4d9c-be54-b67c399b0556-kube-api-access-5vmpq\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:24.143856 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:24.143857    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79jjp\" (UniqueName: \"kubernetes.io/projected/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d-kube-api-access-79jjp\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:05:24.423506 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:24.423476    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l"
Nov 03 08:05:24.423706 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:24.423509    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l" event={"ID":"6fd1cadf-e4be-4d9c-be54-b67c399b0556","Type":"ContainerDied","Data":"67fc1e379a1ecfffb59820885cedc68e10a177b64899ed954aa84df63f53bccd"}
Nov 03 08:05:24.423706 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:24.423563    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67fc1e379a1ecfffb59820885cedc68e10a177b64899ed954aa84df63f53bccd"
Nov 03 08:05:24.429869 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:24.429834    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95" event={"ID":"106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d","Type":"ContainerDied","Data":"e98fa764d99bff105a700c59310e03ae91ac94826c04ef1a669a551362b363d2"}
Nov 03 08:05:24.429994 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:24.429871    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95"
Nov 03 08:05:24.429994 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:24.429887    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e98fa764d99bff105a700c59310e03ae91ac94826c04ef1a669a551362b363d2"
Nov 03 08:05:24.442218 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:24.442194    5070 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 08:05:24.442351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:24.442245    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r" podUID="181a2d26-3317-450e-97be-c0082d47a27a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 08:05:24.911562 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:24.911524    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-l6bvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 08:05:24.912083 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:24.911581    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc" podUID="e33edac1-844e-4caf-bb59-d63291659555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 08:05:25.447287 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:25.447252    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:25.447685 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:25.447663    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:25.448455 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:25.448434    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:05:25.911316 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:25.911283    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-l6bvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 08:05:25.911528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:25.911335    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc" podUID="e33edac1-844e-4caf-bb59-d63291659555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 08:05:26.225248 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:26.225171    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:05:26.911744 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:26.911708    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-l6bvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 08:05:26.911971 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:26.911780    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc" podUID="e33edac1-844e-4caf-bb59-d63291659555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 08:05:27.957898 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:27.957862    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-l6bvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 08:05:27.958486 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:27.958453    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc" podUID="e33edac1-844e-4caf-bb59-d63291659555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 08:05:28.559783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:28.546071    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:28.559968 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:28.559852    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:28.593782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:28.592877    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:28.659777 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:28.658845    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:05:28.660169 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:28.660137    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:28.813858 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:28.813409    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:05:28.913554 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:28.913125    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-l6bvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 08:05:28.913554 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:28.913198    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc" podUID="e33edac1-844e-4caf-bb59-d63291659555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 08:05:29.911162 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:29.911117    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-l6bvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 08:05:29.911664 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:29.911185    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc" podUID="e33edac1-844e-4caf-bb59-d63291659555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 08:05:30.566179 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.566142    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"]
Nov 03 08:05:30.567692 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.567663    5070 cpu_manager.go:414] "containerMap: removing container" podUID="106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d" containerName="revoke-keys"
Nov 03 08:05:30.567692 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.567691    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d" containerName="revoke-keys"
Nov 03 08:05:30.567928 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.567711    5070 cpu_manager.go:414] "containerMap: removing container" podUID="6fd1cadf-e4be-4d9c-be54-b67c399b0556" containerName="revoke-keys"
Nov 03 08:05:30.567928 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.567719    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fd1cadf-e4be-4d9c-be54-b67c399b0556" containerName="revoke-keys"
Nov 03 08:05:30.567928 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.567792    5070 cpu_manager.go:414] "containerMap: removing container" podUID="4397fae5-589b-4a11-9bfc-fefce07a5ab6" containerName="revoke-keys"
Nov 03 08:05:30.567928 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.567803    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4397fae5-589b-4a11-9bfc-fefce07a5ab6" containerName="revoke-keys"
Nov 03 08:05:30.568131 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.567943    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="4397fae5-589b-4a11-9bfc-fefce07a5ab6" containerName="revoke-keys"
Nov 03 08:05:30.568131 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.567967    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d" containerName="revoke-keys"
Nov 03 08:05:30.568131 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.567976    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fd1cadf-e4be-4d9c-be54-b67c399b0556" containerName="revoke-keys"
Nov 03 08:05:30.573953 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.573932    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.591264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.591238    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"]
Nov 03 08:05:30.673162 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.673132    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.673311 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.673199    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.673311 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.673294    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.673832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.673310    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.673832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.673341    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.673832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.673360    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvjz7\" (UniqueName: \"kubernetes.io/projected/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-kube-api-access-lvjz7\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.673832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.673492    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.673832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.673535    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.673832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.673550    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.764533 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.764504    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"]
Nov 03 08:05:30.768711 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.768693    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.774385 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.774356    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.774568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.774444    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.774568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.774473    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.774568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.774525    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.774568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.774550    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvjz7\" (UniqueName: \"kubernetes.io/projected/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-kube-api-access-lvjz7\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.774861 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.774623    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.775319 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.775057    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.775428 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.775384    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.775428 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.775422    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.775546 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.775511    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.776614 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.775865    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.776614 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.776342    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.776614 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.776574    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.776977 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.776959    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.779048 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.779022    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.779574 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.779536    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.779707 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.779682    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"]
Nov 03 08:05:30.786228 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.786198    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvjz7\" (UniqueName: \"kubernetes.io/projected/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-kube-api-access-lvjz7\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.806270 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.806243    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.878904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.877104    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4174a67-8cbf-452f-b382-8efd46377534-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.878904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.877177    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.878904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.877250    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4174a67-8cbf-452f-b382-8efd46377534-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.878904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.877368    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.878904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.877407    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.878904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.877521    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.878904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.877560    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.878904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.877675    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4174a67-8cbf-452f-b382-8efd46377534-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.878904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.877724    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55n5w\" (UniqueName: \"kubernetes.io/projected/d4174a67-8cbf-452f-b382-8efd46377534-kube-api-access-55n5w\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.892722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.892697    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:30.912085 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.912061    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-l6bvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Nov 03 08:05:30.912469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.912120    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc" podUID="e33edac1-844e-4caf-bb59-d63291659555" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Nov 03 08:05:30.978994 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.978868    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.978994 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.978920    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.978994 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.978944    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.978994 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.978979    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4174a67-8cbf-452f-b382-8efd46377534-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.979221 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.979004    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55n5w\" (UniqueName: \"kubernetes.io/projected/d4174a67-8cbf-452f-b382-8efd46377534-kube-api-access-55n5w\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.979221 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.979045    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4174a67-8cbf-452f-b382-8efd46377534-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.979221 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.979083    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.979221 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.979098    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4174a67-8cbf-452f-b382-8efd46377534-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.979221 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.979167    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.979505 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.979488    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.979669 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.979657    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.980299 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.979886    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.980299 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.980066    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.981033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.981010    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4174a67-8cbf-452f-b382-8efd46377534-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.982429 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.982408    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4174a67-8cbf-452f-b382-8efd46377534-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.982519 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.982485    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.987383 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.987332    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4174a67-8cbf-452f-b382-8efd46377534-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:30.988258 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:30.988223    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55n5w\" (UniqueName: \"kubernetes.io/projected/d4174a67-8cbf-452f-b382-8efd46377534-kube-api-access-55n5w\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:31.118825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:31.118798    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:31.184495 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:31.184458    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:05:31.184615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:31.184550    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:05:31.184615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:31.184598    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:05:31.185593 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:31.185485    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"]
Nov 03 08:05:31.418682 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:31.416654    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"]
Nov 03 08:05:31.420663 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:31.420584    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:05:31.420925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:31.420894    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:05:31.421018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:31.420966    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:05:31.612555 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:31.611712    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd" event={"ID":"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b","Type":"ContainerStarted","Data":"4cd7620a2ce00c0da699933e652e7de7f469e1daee40daac8fd4996a105fa128"}
Nov 03 08:05:31.612555 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:31.611857    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd" event={"ID":"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b","Type":"ContainerStarted","Data":"ccebbb56272b2ed2808cd715a2b58f4d90fe106e7ad52df0a9af04ef35db2bcf"}
Nov 03 08:05:31.614954 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:31.614920    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt" event={"ID":"d4174a67-8cbf-452f-b382-8efd46377534","Type":"ContainerStarted","Data":"a25db031f450eb71ce100592f948cee1701b87545c849891ee22aacf0462587f"}
Nov 03 08:05:31.615112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:31.614976    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt" event={"ID":"d4174a67-8cbf-452f-b382-8efd46377534","Type":"ContainerStarted","Data":"8fe9db84b2fe493ba1ea17e540a38cd126bdb7c0cbd5dc5f402c58f6cbc6c9cb"}
Nov 03 08:05:31.648780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:31.639654    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd" podStartSLOduration=1.639635073 podStartE2EDuration="1.639635073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:31.628731685 +0000 UTC m=+729.743195616" watchObservedRunningTime="2026-11-03 08:05:31.639635073 +0000 UTC m=+729.754099003"
Nov 03 08:05:31.676249 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:31.675314    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt" podStartSLOduration=1.6752956430000001 podStartE2EDuration="1.675295643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:31.672653208 +0000 UTC m=+729.787117136" watchObservedRunningTime="2026-11-03 08:05:31.675295643 +0000 UTC m=+729.789759574"
Nov 03 08:05:31.893219 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:31.893179    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:32.119341 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:32.119311    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:32.120672 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:32.120645    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 08:05:32.120843 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:32.120694    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt" podUID="d4174a67-8cbf-452f-b382-8efd46377534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 08:05:32.142241 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:32.142219    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:32.142671 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:32.142558    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:32.144045 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:32.144022    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:05:32.815779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:32.814802    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"]
Nov 03 08:05:32.826928 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:32.826035    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"]
Nov 03 08:05:32.826928 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:32.826099    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:32.832571 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:32.830584    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6bc553\"/\"distinct-2-w2-6bc553-kserve-self-signed-certs\""
Nov 03 08:05:32.832571 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:32.830701    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6bc553\"/\"kube-root-ca.crt\""
Nov 03 08:05:32.832571 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:32.832515    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6bc553\"/\"openshift-service-ca.crt\""
Nov 03 08:05:32.832845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:32.832600    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6bc553\"/\"default-dockercfg-xt4tf\""
Nov 03 08:05:32.910815 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:32.909904    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 08:05:32.910815 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:32.909956    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd" podUID="4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 08:05:32.923937 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:32.923911    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-dshm\") pod \"distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:32.924086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:32.924031    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-model-cache\") pod \"distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:32.924086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:32.924050    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ae55a46-f433-42f5-b29b-dc0c51b87de4-tls-certs\") pod \"distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:32.924086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:32.924081    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-home\") pod \"distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:32.924208 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:32.924119    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx4bt\" (UniqueName: \"kubernetes.io/projected/8ae55a46-f433-42f5-b29b-dc0c51b87de4-kube-api-access-mx4bt\") pod \"distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:32.924208 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:32.924139    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-tmp-dir\") pod \"distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:33.025112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:33.025057    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-home\") pod \"distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:33.026054 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:33.026032    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx4bt\" (UniqueName: \"kubernetes.io/projected/8ae55a46-f433-42f5-b29b-dc0c51b87de4-kube-api-access-mx4bt\") pod \"distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:33.026572 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:33.026545    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-tmp-dir\") pod \"distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:33.027070 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:33.027053    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-dshm\") pod \"distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:33.027736 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:33.025911    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-home\") pod \"distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:33.027888 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:33.026987    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-tmp-dir\") pod \"distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:33.028224 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:33.028194    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-model-cache\") pod \"distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:33.028308 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:33.028241    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ae55a46-f433-42f5-b29b-dc0c51b87de4-tls-certs\") pod \"distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:33.028665 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:33.028639    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-model-cache\") pod \"distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:33.029523 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:33.029476    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-dshm\") pod \"distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:33.034357 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:33.034306    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ae55a46-f433-42f5-b29b-dc0c51b87de4-tls-certs\") pod \"distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:33.036388 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:33.036338    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx4bt\" (UniqueName: \"kubernetes.io/projected/8ae55a46-f433-42f5-b29b-dc0c51b87de4-kube-api-access-mx4bt\") pod \"distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") " pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:33.125785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:33.123157    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 08:05:33.125785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:33.123220    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt" podUID="d4174a67-8cbf-452f-b382-8efd46377534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 08:05:33.141379 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:33.141346    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:33.510534 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:33.510501    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"]
Nov 03 08:05:33.513604 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:05:33.513563    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ae55a46_f433_42f5_b29b_dc0c51b87de4.slice/crio-19b856821075d87252b8b7dd2ecb8e8a714af2689b48c98d744691c1c6781db6 WatchSource:0}: Error finding container 19b856821075d87252b8b7dd2ecb8e8a714af2689b48c98d744691c1c6781db6: Status 404 returned error can't find the container with id 19b856821075d87252b8b7dd2ecb8e8a714af2689b48c98d744691c1c6781db6
Nov 03 08:05:33.643108 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:33.642864    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" event={"ID":"8ae55a46-f433-42f5-b29b-dc0c51b87de4","Type":"ContainerStarted","Data":"b4ef020ec54f355d310476ff5f1bf4b13fcfb9f317b24ca9f693e078617c3daa"}
Nov 03 08:05:33.643108 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:33.642910    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" event={"ID":"8ae55a46-f433-42f5-b29b-dc0c51b87de4","Type":"ContainerStarted","Data":"19b856821075d87252b8b7dd2ecb8e8a714af2689b48c98d744691c1c6781db6"}
Nov 03 08:05:33.717810 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:33.717084    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podStartSLOduration=1.717069271 podStartE2EDuration="1.717069271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:33.69521038 +0000 UTC m=+731.809674312" watchObservedRunningTime="2026-11-03 08:05:33.717069271 +0000 UTC m=+731.831533230"
Nov 03 08:05:33.895784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:33.895057    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 08:05:33.895784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:33.895119    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd" podUID="4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 08:05:34.119840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:34.119803    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 08:05:34.120018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:34.119870    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt" podUID="d4174a67-8cbf-452f-b382-8efd46377534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 08:05:34.141660 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:34.141632    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:34.893157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:34.893118    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 08:05:34.893438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:34.893410    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd" podUID="4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 08:05:35.119263 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:35.119232    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 08:05:35.119499 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:35.119476    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt" podUID="d4174a67-8cbf-452f-b382-8efd46377534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 08:05:35.893783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:35.893143    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 08:05:35.893783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:35.893228    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd" podUID="4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 08:05:36.119655 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:36.119624    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 08:05:36.119880 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:36.119853    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt" podUID="d4174a67-8cbf-452f-b382-8efd46377534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 08:05:36.906801 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:36.906185    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 08:05:36.906801 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:36.906262    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd" podUID="4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 08:05:37.120123 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:37.120087    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 08:05:37.120280 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:37.120150    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt" podUID="d4174a67-8cbf-452f-b382-8efd46377534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 08:05:37.894091 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:37.894053    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Nov 03 08:05:37.894276 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:37.894121    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd" podUID="4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Nov 03 08:05:38.119964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.119922    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Nov 03 08:05:38.120447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.119995    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt" podUID="d4174a67-8cbf-452f-b382-8efd46377534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Nov 03 08:05:38.288186 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.286671    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"]
Nov 03 08:05:38.301501 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.299289    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:38.304371 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.304027    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-bd085d\"/\"default-dockercfg-9w92k\""
Nov 03 08:05:38.304531 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.304202    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-bd085d\"/\"openshift-service-ca.crt\""
Nov 03 08:05:38.304667 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.304277    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-bd085d\"/\"distinct-w6-bd085d-kserve-self-signed-certs\""
Nov 03 08:05:38.304804 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.304334    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-bd085d\"/\"kube-root-ca.crt\""
Nov 03 08:05:38.307596 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.305831    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"]
Nov 03 08:05:38.344863 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.344828    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-tls-certs\") pod \"distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:38.345074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.344889    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-model-cache\") pod \"distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:38.345074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.344948    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-dshm\") pod \"distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:38.345074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.344979    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dk6r\" (UniqueName: \"kubernetes.io/projected/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-kube-api-access-6dk6r\") pod \"distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:38.345074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.345037    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-tmp-dir\") pod \"distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:38.345398 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.345121    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-home\") pod \"distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:38.447783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.446878    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-dshm\") pod \"distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:38.447783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.446915    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dk6r\" (UniqueName: \"kubernetes.io/projected/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-kube-api-access-6dk6r\") pod \"distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:38.447783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.446981    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-tmp-dir\") pod \"distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:38.447783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.447060    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-home\") pod \"distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:38.447783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.447215    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-tls-certs\") pod \"distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:38.447783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.447252    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-model-cache\") pod \"distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:38.447783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.447718    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-model-cache\") pod \"distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:38.447783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.447752    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-home\") pod \"distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:38.448322 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.448278    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-tmp-dir\") pod \"distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:38.450623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.450600    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-dshm\") pod \"distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:38.461176 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.459562    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dk6r\" (UniqueName: \"kubernetes.io/projected/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-kube-api-access-6dk6r\") pod \"distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:38.461726 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.461700    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-tls-certs\") pod \"distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") " pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:38.651219 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.651180    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:38.898322 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.898290    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:38.898791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.898773    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:38.899513 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.899495    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:05:38.930671 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:38.930647    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"]
Nov 03 08:05:38.931129 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:05:38.931104    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd172e36b_89d5_43d3_9ad8_9cc9142d12d0.slice/crio-3f2853b63d867d78f3f1b693b43c5ef47c0db7e43e7cdf9a575dcd07a13f9893 WatchSource:0}: Error finding container 3f2853b63d867d78f3f1b693b43c5ef47c0db7e43e7cdf9a575dcd07a13f9893: Status 404 returned error can't find the container with id 3f2853b63d867d78f3f1b693b43c5ef47c0db7e43e7cdf9a575dcd07a13f9893
Nov 03 08:05:39.123740 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:39.123709    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:39.124177 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:39.124158    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:39.124824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:39.124809    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:05:39.751820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:39.751753    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" event={"ID":"d172e36b-89d5-43d3-9ad8-9cc9142d12d0","Type":"ContainerStarted","Data":"414accb291de910b47a17b89016eed4be600c142606fff6de269ea0f69757017"}
Nov 03 08:05:39.752443 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:39.751833    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" event={"ID":"d172e36b-89d5-43d3-9ad8-9cc9142d12d0","Type":"ContainerStarted","Data":"3f2853b63d867d78f3f1b693b43c5ef47c0db7e43e7cdf9a575dcd07a13f9893"}
Nov 03 08:05:39.770729 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:39.770683    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podStartSLOduration=1.770670345 podStartE2EDuration="1.770670345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:39.766919463 +0000 UTC m=+737.881383394" watchObservedRunningTime="2026-11-03 08:05:39.770670345 +0000 UTC m=+737.885134275"
Nov 03 08:05:40.651660 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.651628    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:40.833989 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.833955    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"]
Nov 03 08:05:40.844136 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.844103    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:40.844722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.844692    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"]
Nov 03 08:05:40.851652 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.851618    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-19bb0a\"/\"distinct-w1-19bb0a-kserve-self-signed-certs\""
Nov 03 08:05:40.882058 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.882028    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs9fz\" (UniqueName: \"kubernetes.io/projected/b6603beb-ef9c-4cd4-96b2-286c95a37353-kube-api-access-rs9fz\") pod \"distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:40.882179 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.882090    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-home\") pod \"distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:40.882252 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.882197    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-dshm\") pod \"distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:40.882510 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.882490    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-tmp-dir\") pod \"distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:40.882826 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.882711    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-model-cache\") pod \"distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:40.882980 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.882953    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6603beb-ef9c-4cd4-96b2-286c95a37353-tls-certs\") pod \"distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:40.985341 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.985249    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-home\") pod \"distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:40.985341 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.985302    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-dshm\") pod \"distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:40.985652 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.985373    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-tmp-dir\") pod \"distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:40.985652 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.985485    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-model-cache\") pod \"distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:40.985652 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.985549    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6603beb-ef9c-4cd4-96b2-286c95a37353-tls-certs\") pod \"distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:40.985882 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.985678    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs9fz\" (UniqueName: \"kubernetes.io/projected/b6603beb-ef9c-4cd4-96b2-286c95a37353-kube-api-access-rs9fz\") pod \"distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:40.986458 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.986372    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-home\") pod \"distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:40.987179 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.987156    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-model-cache\") pod \"distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:40.987356 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.987320    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-tmp-dir\") pod \"distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:40.990020 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.989996    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-dshm\") pod \"distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:40.990146 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:40.990124    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6603beb-ef9c-4cd4-96b2-286c95a37353-tls-certs\") pod \"distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:41.002891 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:41.002862    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs9fz\" (UniqueName: \"kubernetes.io/projected/b6603beb-ef9c-4cd4-96b2-286c95a37353-kube-api-access-rs9fz\") pod \"distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") " pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:41.160303 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:41.160271    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:41.440127 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:41.440086    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"]
Nov 03 08:05:41.767942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:41.767862    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" event={"ID":"b6603beb-ef9c-4cd4-96b2-286c95a37353","Type":"ContainerStarted","Data":"c140ff2083de5c92a6a58adde11782aaef9ac87a67ec52eb0697443528af6df8"}
Nov 03 08:05:41.767942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:41.767903    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" event={"ID":"b6603beb-ef9c-4cd4-96b2-286c95a37353","Type":"ContainerStarted","Data":"252fb4e8a7ff5899c2cd9d9a3e39e64f17dc4ac0e78bb4ae436c307f57473512"}
Nov 03 08:05:41.785317 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:41.785274    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podStartSLOduration=1.785260963 podStartE2EDuration="1.785260963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:41.78269668 +0000 UTC m=+739.897160613" watchObservedRunningTime="2026-11-03 08:05:41.785260963 +0000 UTC m=+739.899724893"
Nov 03 08:05:42.160735 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:42.160701    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:43.141455 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:43.141422    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:43.159903 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:43.159876    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:43.931264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:43.931231    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:45.653581 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.652169    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"]
Nov 03 08:05:45.659743 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.659712    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.667165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.666812    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"]
Nov 03 08:05:45.667165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.667045    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-6bc553-openshift-default-dockercfg-fcnbk\""
Nov 03 08:05:45.828712 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.828675    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.828914 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.828722    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.828914 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.828806    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.828914 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.828866    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.829048 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.828962    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.829048 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.829006    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.829119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.829088    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j749l\" (UniqueName: \"kubernetes.io/projected/8615eb5b-41a4-4c01-b550-c1d3df743738-kube-api-access-j749l\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.829169 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.829154    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.829241 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.829224    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8615eb5b-41a4-4c01-b550-c1d3df743738-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.930523 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.930447    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.930523 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.930494    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.930722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.930541    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j749l\" (UniqueName: \"kubernetes.io/projected/8615eb5b-41a4-4c01-b550-c1d3df743738-kube-api-access-j749l\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.930722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.930579    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.930859 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.930790    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8615eb5b-41a4-4c01-b550-c1d3df743738-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.930941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.930926    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.930997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.930951    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.930997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.930971    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.930997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.930946    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.931148 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.931001    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.931215 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.931191    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.931319 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.931295    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.931399 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.931351    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.931488 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.931472    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8615eb5b-41a4-4c01-b550-c1d3df743738-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.932991 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.932967    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.933359 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.933339    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.936918 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.936892    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.937282 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.937261    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j749l\" (UniqueName: \"kubernetes.io/projected/8615eb5b-41a4-4c01-b550-c1d3df743738-kube-api-access-j749l\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:45.986147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:45.986124    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:46.280617 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:46.280587    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"]
Nov 03 08:05:46.281460 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:05:46.281436    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8615eb5b_41a4_4c01_b550_c1d3df743738.slice/crio-7274782777fa4786380feac996e280a4843abbd42cd6d052018a4219e1394a5c WatchSource:0}: Error finding container 7274782777fa4786380feac996e280a4843abbd42cd6d052018a4219e1394a5c: Status 404 returned error can't find the container with id 7274782777fa4786380feac996e280a4843abbd42cd6d052018a4219e1394a5c
Nov 03 08:05:46.284132 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:46.284066    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:05:46.284279 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:46.284193    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:05:46.284279 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:46.284233    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:05:46.846550 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:46.846511    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r" event={"ID":"8615eb5b-41a4-4c01-b550-c1d3df743738","Type":"ContainerStarted","Data":"48d010e07ea12dd9524c8345df22b4be0189b2c78fbe3bd1151814c129eeeadf"}
Nov 03 08:05:46.847046 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:46.846581    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r" event={"ID":"8615eb5b-41a4-4c01-b550-c1d3df743738","Type":"ContainerStarted","Data":"7274782777fa4786380feac996e280a4843abbd42cd6d052018a4219e1394a5c"}
Nov 03 08:05:46.882053 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:46.881983    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r" podStartSLOduration=1.88195146 podStartE2EDuration="1.88195146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:05:46.875251196 +0000 UTC m=+744.989715129" watchObservedRunningTime="2026-11-03 08:05:46.88195146 +0000 UTC m=+744.996415392"
Nov 03 08:05:46.986877 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:46.986847    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:47.988718 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:47.988685    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 08:05:47.989113 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:47.988731    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r" podUID="8615eb5b-41a4-4c01-b550-c1d3df743738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 08:05:48.651380 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:48.651346    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:48.690921 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:48.690889    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:48.896448 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:48.896417    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:48.987377 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:48.987289    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 08:05:48.987377 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:48.987358    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r" podUID="8615eb5b-41a4-4c01-b550-c1d3df743738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 08:05:49.987213 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:49.987182    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 08:05:49.987610 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:49.987234    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r" podUID="8615eb5b-41a4-4c01-b550-c1d3df743738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 08:05:50.987452 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:50.987418    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 08:05:50.987898 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:50.987471    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r" podUID="8615eb5b-41a4-4c01-b550-c1d3df743738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 08:05:51.181819 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:51.179776    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:51.193658 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:51.193635    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:51.916361 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:51.916326    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:05:51.987892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:51.987854    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 08:05:51.988272 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:51.987922    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r" podUID="8615eb5b-41a4-4c01-b550-c1d3df743738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 08:05:52.050723 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:05:52.050689    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:05:52.986594 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:52.986561    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Nov 03 08:05:52.986823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:52.986618    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r" podUID="8615eb5b-41a4-4c01-b550-c1d3df743738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Nov 03 08:05:53.991052 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:53.991015    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:53.991544 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:53.991388    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:53.991908 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:53.991889    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:05:56.283892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:56.283856    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:05:58.653365 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:58.653321    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:05:58.663797 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:58.663484    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:58.808889 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:58.808842    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:58.825874 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:58.825832    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:58.886788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:58.886605    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:58.924838 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:58.922037    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:05:59.138292 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:59.138006    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:59.611790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:59.610556    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:05:59.611790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:59.610947    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:59.681145 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:59.681100    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:59.893233 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:59.891887    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:05:59.893233 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:59.686668    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:59.893233 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:59.892908    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:05:59.893233 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:59.811851    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:59.893577 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:59.890360    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:59.894218 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:59.686205    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:59.910118 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:59.894956    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:05:59.910118 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:59.828837    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:05:59.910118 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:59.895235    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:05:59.910118 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:59.903121    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:05:59.948716 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:59.948683    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:05:59.985661 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:05:59.975485    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:06:00.075845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.055577    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:06:00.084585 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.084558    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:06:00.135017 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.134986    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:06:00.304746 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.304666    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"]
Nov 03 08:06:00.311973 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.311950    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.314997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.314973    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 08:06:00.315258 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.315232    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vk5zp\""
Nov 03 08:06:00.315377 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.315271    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 08:06:00.315377 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.315320    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 08:06:00.315702 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.315681    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"]
Nov 03 08:06:00.375426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.374511    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26r8z\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.375426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.374598    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26r8z\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.375426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.374635    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvcnm\" (UniqueName: \"kubernetes.io/projected/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-kube-api-access-bvcnm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26r8z\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.375426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.374670    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26r8z\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.375426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.374732    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26r8z\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.375426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.374780    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26r8z\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.375426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.374869    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26r8z\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.476088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.476049    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvcnm\" (UniqueName: \"kubernetes.io/projected/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-kube-api-access-bvcnm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26r8z\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.476278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.476098    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26r8z\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.476278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.476153    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26r8z\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.476278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.476176    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26r8z\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.476437 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.476276    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26r8z\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.476490 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.476465    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26r8z\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.476547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.476534    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26r8z\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.477243 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.477213    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26r8z\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.479038 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.479004    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26r8z\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.479173 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.479073    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26r8z\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.479354 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.479325    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26r8z\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.479542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.479519    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26r8z\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.507214 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.507179    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26r8z\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.507671 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.507646    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvcnm\" (UniqueName: \"kubernetes.io/projected/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-kube-api-access-bvcnm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-26r8z\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.629645 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.629613    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:00.921041 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:06:00.921006    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c0eb74_3811_4db9_b783_b1b9a14ee7b5.slice/crio-7dd0d2193469fe9eabeba8d93489ae89a1204177225064ef18cabfeeac48939d WatchSource:0}: Error finding container 7dd0d2193469fe9eabeba8d93489ae89a1204177225064ef18cabfeeac48939d: Status 404 returned error can't find the container with id 7dd0d2193469fe9eabeba8d93489ae89a1204177225064ef18cabfeeac48939d
Nov 03 08:06:00.921727 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:00.921699    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"]
Nov 03 08:06:01.167296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:01.167210    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z" event={"ID":"83c0eb74-3811-4db9-b783-b1b9a14ee7b5","Type":"ContainerStarted","Data":"7dd0d2193469fe9eabeba8d93489ae89a1204177225064ef18cabfeeac48939d"}
Nov 03 08:06:04.182876 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:04.182846    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:04.183141 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:04.182904    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:05.200740 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:05.200698    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z" event={"ID":"83c0eb74-3811-4db9-b783-b1b9a14ee7b5","Type":"ContainerStarted","Data":"f25ba54996fdbc7c20ebec38667dc32d5f81de3f783abeffffd12150c17e991d"}
Nov 03 08:06:05.201216 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:05.200889    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:05.202426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:05.202405    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:06:05.218171 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:05.218125    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z" podStartSLOduration=1.959631836 podStartE2EDuration="5.218112165s" totalImagesPullingTime="3.258480329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:05.216718908 +0000 UTC m=+763.331182839" watchObservedRunningTime="2026-11-03 08:06:05.218112165 +0000 UTC m=+763.332576099"
Nov 03 08:06:06.756347 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:06.756316    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h"]
Nov 03 08:06:06.760678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:06.760657    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h"
Nov 03 08:06:06.763852 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:06.763829    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:06:06.764092 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:06.764058    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:06:06.764092 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:06.764070    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:06:06.764265 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:06.763951    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:06:06.768067 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:06.768045    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h"]
Nov 03 08:06:06.859682 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:06.859650    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52b6p\" (UniqueName: \"kubernetes.io/projected/01e9078f-803f-4c08-8608-5f359bb8e3e4-kube-api-access-52b6p\") pod \"maas-api-revoke-sub-9ac77225-qhd8h\" (UID: \"01e9078f-803f-4c08-8608-5f359bb8e3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h"
Nov 03 08:06:06.859865 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:06.859843    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e9078f-803f-4c08-8608-5f359bb8e3e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ac77225-qhd8h\" (UID: \"01e9078f-803f-4c08-8608-5f359bb8e3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h"
Nov 03 08:06:06.859934 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:06.859873    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e9078f-803f-4c08-8608-5f359bb8e3e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ac77225-qhd8h\" (UID: \"01e9078f-803f-4c08-8608-5f359bb8e3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h"
Nov 03 08:06:06.961047 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:06.961017    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e9078f-803f-4c08-8608-5f359bb8e3e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ac77225-qhd8h\" (UID: \"01e9078f-803f-4c08-8608-5f359bb8e3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h"
Nov 03 08:06:06.961165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:06.961050    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e9078f-803f-4c08-8608-5f359bb8e3e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ac77225-qhd8h\" (UID: \"01e9078f-803f-4c08-8608-5f359bb8e3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h"
Nov 03 08:06:06.961165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:06.961155    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52b6p\" (UniqueName: \"kubernetes.io/projected/01e9078f-803f-4c08-8608-5f359bb8e3e4-kube-api-access-52b6p\") pod \"maas-api-revoke-sub-9ac77225-qhd8h\" (UID: \"01e9078f-803f-4c08-8608-5f359bb8e3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h"
Nov 03 08:06:06.961708 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:06.961688    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e9078f-803f-4c08-8608-5f359bb8e3e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ac77225-qhd8h\" (UID: \"01e9078f-803f-4c08-8608-5f359bb8e3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h"
Nov 03 08:06:06.963616 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:06.963594    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e9078f-803f-4c08-8608-5f359bb8e3e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ac77225-qhd8h\" (UID: \"01e9078f-803f-4c08-8608-5f359bb8e3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h"
Nov 03 08:06:06.970330 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:06.970308    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52b6p\" (UniqueName: \"kubernetes.io/projected/01e9078f-803f-4c08-8608-5f359bb8e3e4-kube-api-access-52b6p\") pod \"maas-api-revoke-sub-9ac77225-qhd8h\" (UID: \"01e9078f-803f-4c08-8608-5f359bb8e3e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h"
Nov 03 08:06:07.077907 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:07.077852    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h"
Nov 03 08:06:07.356949 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:06:07.356922    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01e9078f_803f_4c08_8608_5f359bb8e3e4.slice/crio-ccbf68018f0e97d025b6366b040eb710fa463e454a9c2e1edee32077c814f57d WatchSource:0}: Error finding container ccbf68018f0e97d025b6366b040eb710fa463e454a9c2e1edee32077c814f57d: Status 404 returned error can't find the container with id ccbf68018f0e97d025b6366b040eb710fa463e454a9c2e1edee32077c814f57d
Nov 03 08:06:07.357863 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:07.357836    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h"]
Nov 03 08:06:08.225546 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:08.225504    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h" event={"ID":"01e9078f-803f-4c08-8608-5f359bb8e3e4","Type":"ContainerStarted","Data":"9beef623c3e4a2bb1b9d609c622ed186477965a0a040890d4d54774bc44dd571"}
Nov 03 08:06:08.226000 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:08.225579    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h" event={"ID":"01e9078f-803f-4c08-8608-5f359bb8e3e4","Type":"ContainerStarted","Data":"ccbf68018f0e97d025b6366b040eb710fa463e454a9c2e1edee32077c814f57d"}
Nov 03 08:06:08.243267 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:08.243225    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h" podStartSLOduration=2.24321255 podStartE2EDuration="2.24321255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:08.238795575 +0000 UTC m=+766.353259506" watchObservedRunningTime="2026-11-03 08:06:08.24321255 +0000 UTC m=+766.357676544"
Nov 03 08:06:09.243064 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:09.243023    5070 generic.go:356] "Generic (PLEG): container finished" podID="01e9078f-803f-4c08-8608-5f359bb8e3e4" containerID="9beef623c3e4a2bb1b9d609c622ed186477965a0a040890d4d54774bc44dd571" exitCode=0
Nov 03 08:06:09.243541 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:09.243105    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h" event={"ID":"01e9078f-803f-4c08-8608-5f359bb8e3e4","Type":"ContainerDied","Data":"9beef623c3e4a2bb1b9d609c622ed186477965a0a040890d4d54774bc44dd571"}
Nov 03 08:06:10.407553 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:10.407530    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h"
Nov 03 08:06:10.506411 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:10.506344    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01e9078f-803f-4c08-8608-5f359bb8e3e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e9078f-803f-4c08-8608-5f359bb8e3e4-maas-api-openshift-ca\") pod \"01e9078f-803f-4c08-8608-5f359bb8e3e4\" (UID: \"01e9078f-803f-4c08-8608-5f359bb8e3e4\") "
Nov 03 08:06:10.506544 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:10.506451    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01e9078f-803f-4c08-8608-5f359bb8e3e4-kube-api-access-52b6p\" (UniqueName: \"kubernetes.io/projected/01e9078f-803f-4c08-8608-5f359bb8e3e4-kube-api-access-52b6p\") pod \"01e9078f-803f-4c08-8608-5f359bb8e3e4\" (UID: \"01e9078f-803f-4c08-8608-5f359bb8e3e4\") "
Nov 03 08:06:10.506544 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:10.506523    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01e9078f-803f-4c08-8608-5f359bb8e3e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e9078f-803f-4c08-8608-5f359bb8e3e4-maas-api-xks-ca\") pod \"01e9078f-803f-4c08-8608-5f359bb8e3e4\" (UID: \"01e9078f-803f-4c08-8608-5f359bb8e3e4\") "
Nov 03 08:06:10.506744 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:10.506716    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01e9078f-803f-4c08-8608-5f359bb8e3e4-maas-api-openshift-ca" pod "01e9078f-803f-4c08-8608-5f359bb8e3e4" (UID: "01e9078f-803f-4c08-8608-5f359bb8e3e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:06:10.506879 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:10.506863    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01e9078f-803f-4c08-8608-5f359bb8e3e4-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:10.508971 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:10.508937    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01e9078f-803f-4c08-8608-5f359bb8e3e4-maas-api-xks-ca" pod "01e9078f-803f-4c08-8608-5f359bb8e3e4" (UID: "01e9078f-803f-4c08-8608-5f359bb8e3e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:06:10.509049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:10.508996    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01e9078f-803f-4c08-8608-5f359bb8e3e4-kube-api-access-52b6p" pod "01e9078f-803f-4c08-8608-5f359bb8e3e4" (UID: "01e9078f-803f-4c08-8608-5f359bb8e3e4"). InnerVolumeSpecName "kube-api-access-52b6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:06:10.607749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:10.607723    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52b6p\" (UniqueName: \"kubernetes.io/projected/01e9078f-803f-4c08-8608-5f359bb8e3e4-kube-api-access-52b6p\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:10.607857 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:10.607749    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01e9078f-803f-4c08-8608-5f359bb8e3e4-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:11.258792 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:11.258749    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h"
Nov 03 08:06:11.258972 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:11.258797    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h" event={"ID":"01e9078f-803f-4c08-8608-5f359bb8e3e4","Type":"ContainerDied","Data":"ccbf68018f0e97d025b6366b040eb710fa463e454a9c2e1edee32077c814f57d"}
Nov 03 08:06:11.258972 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:11.258843    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccbf68018f0e97d025b6366b040eb710fa463e454a9c2e1edee32077c814f57d"
Nov 03 08:06:15.501966 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.501928    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"]
Nov 03 08:06:15.502528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.502514    5070 cpu_manager.go:414] "containerMap: removing container" podUID="01e9078f-803f-4c08-8608-5f359bb8e3e4" containerName="revoke-keys"
Nov 03 08:06:15.502590 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.502529    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01e9078f-803f-4c08-8608-5f359bb8e3e4" containerName="revoke-keys"
Nov 03 08:06:15.502690 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.502678    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="01e9078f-803f-4c08-8608-5f359bb8e3e4" containerName="revoke-keys"
Nov 03 08:06:15.506740 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.506717    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.533108 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.533078    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"]
Nov 03 08:06:15.656722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.656689    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"]
Nov 03 08:06:15.661498 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.661475    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.670260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.670230    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.670383 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.670305    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.670447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.670381    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.670447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.670410    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.670552 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.670454    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7d0e21-8828-4ebd-a135-068d713fe575-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.670552 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.670514    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.670654 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.670568    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.670654 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.670608    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbqlx\" (UniqueName: \"kubernetes.io/projected/2a7d0e21-8828-4ebd-a135-068d713fe575-kube-api-access-gbqlx\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.670816 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.670779    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.703110 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.703074    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"]
Nov 03 08:06:15.772494 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.772421    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.772494 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.772487    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.772651 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.772558    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.772651 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.772584    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.772651 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.772611    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.772651 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.772633    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.772901 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.772736    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.772973 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.772892    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.772973 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.772926    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.772973 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.772930    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.773086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.773032    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.773086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.773057    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.773086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.773083    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfxb8\" (UniqueName: \"kubernetes.io/projected/56d6e143-5bb7-42a8-a5e6-0f14214b7592-kube-api-access-tfxb8\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.773241 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.773106    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7d0e21-8828-4ebd-a135-068d713fe575-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.773241 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.773131    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.773241 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.773177    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.773241 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.773216    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.773445 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.773273    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.773445 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.773306    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.773445 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.773338    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbqlx\" (UniqueName: \"kubernetes.io/projected/2a7d0e21-8828-4ebd-a135-068d713fe575-kube-api-access-gbqlx\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.773633 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.773561    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.773693 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.773647    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.773950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.773923    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7d0e21-8828-4ebd-a135-068d713fe575-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.775448 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.775425    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.775582 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.775563    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.781367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.781348    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.781484 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.781467    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbqlx\" (UniqueName: \"kubernetes.io/projected/2a7d0e21-8828-4ebd-a135-068d713fe575-kube-api-access-gbqlx\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.845713 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.845688    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:15.874401 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.874361    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfxb8\" (UniqueName: \"kubernetes.io/projected/56d6e143-5bb7-42a8-a5e6-0f14214b7592-kube-api-access-tfxb8\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.874510 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.874407    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.874510 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.874475    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.874709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.874552    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.874709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.874635    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.874709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.874657    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.874709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.874687    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.874709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.874704    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.875051 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.874753    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.875051 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.874805    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-data\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.875159 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.875093    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-workload-certs\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.875231 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.875208    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-credential-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.875288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.875277    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-workload-socket\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.875608 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.875587    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istiod-ca-cert\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.877256 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.877191    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-envoy\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.877543 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.877523    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-podinfo\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.881051 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.881027    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-token\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.881250 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.881231    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfxb8\" (UniqueName: \"kubernetes.io/projected/56d6e143-5bb7-42a8-a5e6-0f14214b7592-kube-api-access-tfxb8\") pod \"e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") " pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:15.981295 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:15.981262    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:16.127701 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:06:16.127672    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7d0e21_8828_4ebd_a135_068d713fe575.slice/crio-83cc0229870dcfe5d1463b5c520d2899734ec8b7c0a4f2b3ac84fb94225b67fd WatchSource:0}: Error finding container 83cc0229870dcfe5d1463b5c520d2899734ec8b7c0a4f2b3ac84fb94225b67fd: Status 404 returned error can't find the container with id 83cc0229870dcfe5d1463b5c520d2899734ec8b7c0a4f2b3ac84fb94225b67fd
Nov 03 08:06:16.129445 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.129420    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"]
Nov 03 08:06:16.130543 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.130512    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:16.130616 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.130570    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:16.130616 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.130603    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:16.262455 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.262426    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"]
Nov 03 08:06:16.263112 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:06:16.263086    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d6e143_5bb7_42a8_a5e6_0f14214b7592.slice/crio-8220b41807cf5628e84077e51b32779e4b74ece2d1834c777dc6c7124e254707 WatchSource:0}: Error finding container 8220b41807cf5628e84077e51b32779e4b74ece2d1834c777dc6c7124e254707: Status 404 returned error can't find the container with id 8220b41807cf5628e84077e51b32779e4b74ece2d1834c777dc6c7124e254707
Nov 03 08:06:16.266165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.266130    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:16.266239 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.266202    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:16.266285 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.266236    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:16.299943 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.299911    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5" event={"ID":"2a7d0e21-8828-4ebd-a135-068d713fe575","Type":"ContainerStarted","Data":"a68fa3625203c296044b59b085178ebaa46e9f3b35bab2fd7b891a166a3236b9"}
Nov 03 08:06:16.300081 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.299970    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5" event={"ID":"2a7d0e21-8828-4ebd-a135-068d713fe575","Type":"ContainerStarted","Data":"83cc0229870dcfe5d1463b5c520d2899734ec8b7c0a4f2b3ac84fb94225b67fd"}
Nov 03 08:06:16.301573 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.301547    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt" event={"ID":"56d6e143-5bb7-42a8-a5e6-0f14214b7592","Type":"ContainerStarted","Data":"8220b41807cf5628e84077e51b32779e4b74ece2d1834c777dc6c7124e254707"}
Nov 03 08:06:16.344205 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.341325    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5" podStartSLOduration=1.341310482 podStartE2EDuration="1.341310482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:16.323264909 +0000 UTC m=+774.437728843" watchObservedRunningTime="2026-11-03 08:06:16.341310482 +0000 UTC m=+774.455774412"
Nov 03 08:06:16.422832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.422793    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b742d02c-kj9hm"]
Nov 03 08:06:16.427233 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.427214    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b742d02c-kj9hm"
Nov 03 08:06:16.430661 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.430632    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:06:16.430812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.430675    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:06:16.435559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.435535    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b742d02c-kj9hm"]
Nov 03 08:06:16.454590 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.453606    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:06:16.454590 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.453680    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:06:16.588066 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.587973    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmxll\" (UniqueName: \"kubernetes.io/projected/53da5571-e475-4695-9764-956e91f25f21-kube-api-access-wmxll\") pod \"maas-api-revoke-sub-b742d02c-kj9hm\" (UID: \"53da5571-e475-4695-9764-956e91f25f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b742d02c-kj9hm"
Nov 03 08:06:16.588066 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.588022    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53da5571-e475-4695-9764-956e91f25f21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b742d02c-kj9hm\" (UID: \"53da5571-e475-4695-9764-956e91f25f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b742d02c-kj9hm"
Nov 03 08:06:16.588449 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.588204    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53da5571-e475-4695-9764-956e91f25f21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b742d02c-kj9hm\" (UID: \"53da5571-e475-4695-9764-956e91f25f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b742d02c-kj9hm"
Nov 03 08:06:16.690131 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.690099    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53da5571-e475-4695-9764-956e91f25f21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b742d02c-kj9hm\" (UID: \"53da5571-e475-4695-9764-956e91f25f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b742d02c-kj9hm"
Nov 03 08:06:16.690336 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.690268    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmxll\" (UniqueName: \"kubernetes.io/projected/53da5571-e475-4695-9764-956e91f25f21-kube-api-access-wmxll\") pod \"maas-api-revoke-sub-b742d02c-kj9hm\" (UID: \"53da5571-e475-4695-9764-956e91f25f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b742d02c-kj9hm"
Nov 03 08:06:16.690336 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.690310    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53da5571-e475-4695-9764-956e91f25f21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b742d02c-kj9hm\" (UID: \"53da5571-e475-4695-9764-956e91f25f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b742d02c-kj9hm"
Nov 03 08:06:16.691067 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.691042    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53da5571-e475-4695-9764-956e91f25f21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b742d02c-kj9hm\" (UID: \"53da5571-e475-4695-9764-956e91f25f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b742d02c-kj9hm"
Nov 03 08:06:16.693156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.693124    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53da5571-e475-4695-9764-956e91f25f21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b742d02c-kj9hm\" (UID: \"53da5571-e475-4695-9764-956e91f25f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b742d02c-kj9hm"
Nov 03 08:06:16.697629 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.697607    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmxll\" (UniqueName: \"kubernetes.io/projected/53da5571-e475-4695-9764-956e91f25f21-kube-api-access-wmxll\") pod \"maas-api-revoke-sub-b742d02c-kj9hm\" (UID: \"53da5571-e475-4695-9764-956e91f25f21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b742d02c-kj9hm"
Nov 03 08:06:16.752396 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.752366    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b742d02c-kj9hm"
Nov 03 08:06:16.846132 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:16.846060    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:17.034667 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:06:17.034640    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53da5571_e475_4695_9764_956e91f25f21.slice/crio-1b2a5900ebbafcca95a8f3f2fd4168c6916619675b342a4e014ab85a2757b74b WatchSource:0}: Error finding container 1b2a5900ebbafcca95a8f3f2fd4168c6916619675b342a4e014ab85a2757b74b: Status 404 returned error can't find the container with id 1b2a5900ebbafcca95a8f3f2fd4168c6916619675b342a4e014ab85a2757b74b
Nov 03 08:06:17.035962 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:17.035934    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b742d02c-kj9hm"]
Nov 03 08:06:17.311378 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:17.311331    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt" event={"ID":"56d6e143-5bb7-42a8-a5e6-0f14214b7592","Type":"ContainerStarted","Data":"ac0bcc62511738a8bf8144be94bda9ef0d8664b9bea75a800ef8c4b0d9876744"}
Nov 03 08:06:17.312679 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:17.312649    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b742d02c-kj9hm" event={"ID":"53da5571-e475-4695-9764-956e91f25f21","Type":"ContainerStarted","Data":"1b2a5900ebbafcca95a8f3f2fd4168c6916619675b342a4e014ab85a2757b74b"}
Nov 03 08:06:17.331424 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:17.331378    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt" podStartSLOduration=2.331367038 podStartE2EDuration="2.331367038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:17.326654521 +0000 UTC m=+775.441118454" watchObservedRunningTime="2026-11-03 08:06:17.331367038 +0000 UTC m=+775.445830969"
Nov 03 08:06:17.847600 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:17.847572    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 08:06:17.848051 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:17.848026    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5" podUID="2a7d0e21-8828-4ebd-a135-068d713fe575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 08:06:17.990223 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:17.984104    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:17.996779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:17.993924    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 08:06:17.996779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:17.994005    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt" podUID="56d6e143-5bb7-42a8-a5e6-0f14214b7592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 08:06:18.327820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:18.327742    5070 generic.go:356] "Generic (PLEG): container finished" podID="53da5571-e475-4695-9764-956e91f25f21" containerID="8b8343f74dbe597bc31a34d88260e3089465dfad99f8083d304909bb83979707" exitCode=0
Nov 03 08:06:18.329330 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:18.329298    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b742d02c-kj9hm" event={"ID":"53da5571-e475-4695-9764-956e91f25f21","Type":"ContainerDied","Data":"8b8343f74dbe597bc31a34d88260e3089465dfad99f8083d304909bb83979707"}
Nov 03 08:06:18.846399 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:18.846368    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 08:06:18.846575 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:18.846431    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5" podUID="2a7d0e21-8828-4ebd-a135-068d713fe575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 08:06:18.994782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:18.994096    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 08:06:18.994782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:18.994175    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt" podUID="56d6e143-5bb7-42a8-a5e6-0f14214b7592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 08:06:19.691783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:19.691170    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b742d02c-kj9hm"
Nov 03 08:06:19.752065 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:19.752028    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53da5571-e475-4695-9764-956e91f25f21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53da5571-e475-4695-9764-956e91f25f21-maas-api-xks-ca\") pod \"53da5571-e475-4695-9764-956e91f25f21\" (UID: \"53da5571-e475-4695-9764-956e91f25f21\") "
Nov 03 08:06:19.752493 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:19.752472    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53da5571-e475-4695-9764-956e91f25f21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53da5571-e475-4695-9764-956e91f25f21-maas-api-openshift-ca\") pod \"53da5571-e475-4695-9764-956e91f25f21\" (UID: \"53da5571-e475-4695-9764-956e91f25f21\") "
Nov 03 08:06:19.752614 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:19.752598    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53da5571-e475-4695-9764-956e91f25f21-kube-api-access-wmxll\" (UniqueName: \"kubernetes.io/projected/53da5571-e475-4695-9764-956e91f25f21-kube-api-access-wmxll\") pod \"53da5571-e475-4695-9764-956e91f25f21\" (UID: \"53da5571-e475-4695-9764-956e91f25f21\") "
Nov 03 08:06:19.754091 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:19.754026    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53da5571-e475-4695-9764-956e91f25f21-maas-api-openshift-ca" pod "53da5571-e475-4695-9764-956e91f25f21" (UID: "53da5571-e475-4695-9764-956e91f25f21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:06:19.755385 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:19.755355    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53da5571-e475-4695-9764-956e91f25f21-kube-api-access-wmxll" pod "53da5571-e475-4695-9764-956e91f25f21" (UID: "53da5571-e475-4695-9764-956e91f25f21"). InnerVolumeSpecName "kube-api-access-wmxll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:06:19.824009 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:19.820995    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53da5571-e475-4695-9764-956e91f25f21-maas-api-xks-ca" pod "53da5571-e475-4695-9764-956e91f25f21" (UID: "53da5571-e475-4695-9764-956e91f25f21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:06:19.846517 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:19.846478    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 08:06:19.846681 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:19.846541    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5" podUID="2a7d0e21-8828-4ebd-a135-068d713fe575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 08:06:19.853877 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:19.853848    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmxll\" (UniqueName: \"kubernetes.io/projected/53da5571-e475-4695-9764-956e91f25f21-kube-api-access-wmxll\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:19.853877 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:19.853876    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53da5571-e475-4695-9764-956e91f25f21-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:19.854093 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:19.853887    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53da5571-e475-4695-9764-956e91f25f21-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:20.012845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.003830    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 08:06:20.012845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.003908    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt" podUID="56d6e143-5bb7-42a8-a5e6-0f14214b7592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 08:06:20.345165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.345079    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b742d02c-kj9hm"
Nov 03 08:06:20.345302 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.345078    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b742d02c-kj9hm" event={"ID":"53da5571-e475-4695-9764-956e91f25f21","Type":"ContainerDied","Data":"1b2a5900ebbafcca95a8f3f2fd4168c6916619675b342a4e014ab85a2757b74b"}
Nov 03 08:06:20.345302 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.345206    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b2a5900ebbafcca95a8f3f2fd4168c6916619675b342a4e014ab85a2757b74b"
Nov 03 08:06:20.628724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.627082    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv"]
Nov 03 08:06:20.628724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.627724    5070 cpu_manager.go:414] "containerMap: removing container" podUID="53da5571-e475-4695-9764-956e91f25f21" containerName="revoke-keys"
Nov 03 08:06:20.628724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.627737    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53da5571-e475-4695-9764-956e91f25f21" containerName="revoke-keys"
Nov 03 08:06:20.628724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.627872    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="53da5571-e475-4695-9764-956e91f25f21" containerName="revoke-keys"
Nov 03 08:06:20.632019 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.631988    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv"
Nov 03 08:06:20.636207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.636181    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:06:20.636367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.636353    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:06:20.636467 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.636454    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:06:20.636555 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.636542    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:06:20.640221 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.640199    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv"]
Nov 03 08:06:20.717779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.717576    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3846835c-7mhrv\" (UID: \"dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv"
Nov 03 08:06:20.717982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.717952    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pckvd\" (UniqueName: \"kubernetes.io/projected/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a-kube-api-access-pckvd\") pod \"maas-api-revoke-sub-3846835c-7mhrv\" (UID: \"dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv"
Nov 03 08:06:20.718067 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.718017    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3846835c-7mhrv\" (UID: \"dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv"
Nov 03 08:06:20.819896 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.819864    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3846835c-7mhrv\" (UID: \"dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv"
Nov 03 08:06:20.820088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.819926    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pckvd\" (UniqueName: \"kubernetes.io/projected/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a-kube-api-access-pckvd\") pod \"maas-api-revoke-sub-3846835c-7mhrv\" (UID: \"dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv"
Nov 03 08:06:20.820088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.819972    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3846835c-7mhrv\" (UID: \"dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv"
Nov 03 08:06:20.820722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.820695    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3846835c-7mhrv\" (UID: \"dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv"
Nov 03 08:06:20.822628 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.822605    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3846835c-7mhrv\" (UID: \"dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv"
Nov 03 08:06:20.828334 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.828313    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pckvd\" (UniqueName: \"kubernetes.io/projected/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a-kube-api-access-pckvd\") pod \"maas-api-revoke-sub-3846835c-7mhrv\" (UID: \"dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv"
Nov 03 08:06:20.846591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.846138    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 08:06:20.846591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.846198    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5" podUID="2a7d0e21-8828-4ebd-a135-068d713fe575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 08:06:20.952287 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:20.952194    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv"
Nov 03 08:06:21.016468 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:21.014169    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 08:06:21.016468 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:21.014241    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt" podUID="56d6e143-5bb7-42a8-a5e6-0f14214b7592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 08:06:21.348960 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:06:21.348912    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbdf6a0a_71b0_4ac6_a75a_0df538e27d5a.slice/crio-0bdbb1d8c1fd818e6adec0d5cc5a372a8c148f3d73e162b6f59684046c56da23 WatchSource:0}: Error finding container 0bdbb1d8c1fd818e6adec0d5cc5a372a8c148f3d73e162b6f59684046c56da23: Status 404 returned error can't find the container with id 0bdbb1d8c1fd818e6adec0d5cc5a372a8c148f3d73e162b6f59684046c56da23
Nov 03 08:06:21.349104 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:21.348978    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv"]
Nov 03 08:06:21.846858 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:21.846819    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 08:06:21.847010 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:21.846880    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5" podUID="2a7d0e21-8828-4ebd-a135-068d713fe575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 08:06:21.988785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:21.982046    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 08:06:21.988785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:21.982101    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt" podUID="56d6e143-5bb7-42a8-a5e6-0f14214b7592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 08:06:22.052021 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:06:22.051960    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:06:22.412404 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:22.412355    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv" event={"ID":"dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a","Type":"ContainerStarted","Data":"e21c0c531633ae4b960b8e773afd955ec5ecae6e32cf410900cf829e113b4280"}
Nov 03 08:06:22.412680 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:22.412429    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv" event={"ID":"dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a","Type":"ContainerStarted","Data":"0bdbb1d8c1fd818e6adec0d5cc5a372a8c148f3d73e162b6f59684046c56da23"}
Nov 03 08:06:22.428225 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:22.428184    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv" podStartSLOduration=2.428170589 podStartE2EDuration="2.428170589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:22.42571308 +0000 UTC m=+780.540177011" watchObservedRunningTime="2026-11-03 08:06:22.428170589 +0000 UTC m=+780.542634517"
Nov 03 08:06:22.846767 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:22.846660    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 08:06:22.846767 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:22.846716    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5" podUID="2a7d0e21-8828-4ebd-a135-068d713fe575" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 08:06:22.982695 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:22.982655    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 08:06:22.982887 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:22.982728    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt" podUID="56d6e143-5bb7-42a8-a5e6-0f14214b7592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 08:06:23.425283 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:23.425243    5070 generic.go:356] "Generic (PLEG): container finished" podID="dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a" containerID="e21c0c531633ae4b960b8e773afd955ec5ecae6e32cf410900cf829e113b4280" exitCode=0
Nov 03 08:06:23.425703 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:23.425338    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv" event={"ID":"dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a","Type":"ContainerDied","Data":"e21c0c531633ae4b960b8e773afd955ec5ecae6e32cf410900cf829e113b4280"}
Nov 03 08:06:23.946187 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:23.946144    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:23.946606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:23.946580    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:23.947716 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:23.947692    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:06:24.158126 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:24.158094    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:24.158689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:24.158663    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:24.159557 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:24.159535    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:06:24.603868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:24.603841    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv"
Nov 03 08:06:24.683688 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:24.683604    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a-maas-api-xks-ca\") pod \"dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a\" (UID: \"dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a\") "
Nov 03 08:06:24.683884 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:24.683728    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a-maas-api-openshift-ca\") pod \"dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a\" (UID: \"dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a\") "
Nov 03 08:06:24.683884 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:24.683776    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a-kube-api-access-pckvd\" (UniqueName: \"kubernetes.io/projected/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a-kube-api-access-pckvd\") pod \"dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a\" (UID: \"dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a\") "
Nov 03 08:06:24.685637 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:24.685162    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a-maas-api-openshift-ca" pod "dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a" (UID: "dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:06:24.691279 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:24.691250    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a-kube-api-access-pckvd" pod "dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a" (UID: "dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a"). InnerVolumeSpecName "kube-api-access-pckvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:06:24.693021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:24.692996    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a-maas-api-xks-ca" pod "dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a" (UID: "dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:06:24.786263 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:24.786226    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:24.786263 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:24.786257    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:24.786263 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:24.786268    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pckvd\" (UniqueName: \"kubernetes.io/projected/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a-kube-api-access-pckvd\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:25.442390 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:25.442362    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv"
Nov 03 08:06:25.442581 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:25.442393    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv" event={"ID":"dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a","Type":"ContainerDied","Data":"0bdbb1d8c1fd818e6adec0d5cc5a372a8c148f3d73e162b6f59684046c56da23"}
Nov 03 08:06:25.442581 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:25.442445    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bdbb1d8c1fd818e6adec0d5cc5a372a8c148f3d73e162b6f59684046c56da23"
Nov 03 08:06:27.540806 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:27.535989    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:06:27.930783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:27.914832    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:06:27.979783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:27.976860    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:28.076840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:28.062863    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:28.090834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:28.088845    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:28.102782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:28.101836    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:06:28.535898 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:28.535848    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:28.537302 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:28.537271    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:06:28.537857 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:28.537834    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:06:28.671482 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:28.671263    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:28.709786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:28.680119    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:06:28.919510 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:28.919372    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:28.921723 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:28.921693    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:06:28.921976 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:28.921949    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:06:29.091485 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:29.091452    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:06:32.439654 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.439620    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6"]
Nov 03 08:06:32.440312 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.440279    5070 cpu_manager.go:414] "containerMap: removing container" podUID="dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a" containerName="revoke-keys"
Nov 03 08:06:32.440312 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.440296    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a" containerName="revoke-keys"
Nov 03 08:06:32.440446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.440436    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a" containerName="revoke-keys"
Nov 03 08:06:32.444476 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.444456    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6"
Nov 03 08:06:32.447505 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.447474    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:06:32.447815 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.447798    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:06:32.448026 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.448009    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:06:32.448192 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.448179    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:06:32.452711 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.452677    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6"]
Nov 03 08:06:32.512825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.512789    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qsrw\" (UniqueName: \"kubernetes.io/projected/323b9bca-191d-4d7e-8b2b-a41ec4be28a8-kube-api-access-7qsrw\") pod \"maas-api-revoke-sub-598c7ad0-zqkh6\" (UID: \"323b9bca-191d-4d7e-8b2b-a41ec4be28a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6"
Nov 03 08:06:32.512981 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.512965    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323b9bca-191d-4d7e-8b2b-a41ec4be28a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-598c7ad0-zqkh6\" (UID: \"323b9bca-191d-4d7e-8b2b-a41ec4be28a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6"
Nov 03 08:06:32.513156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.513141    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323b9bca-191d-4d7e-8b2b-a41ec4be28a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-598c7ad0-zqkh6\" (UID: \"323b9bca-191d-4d7e-8b2b-a41ec4be28a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6"
Nov 03 08:06:32.596329 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.596296    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f"]
Nov 03 08:06:32.600867 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.600850    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f"
Nov 03 08:06:32.609004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.608983    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f"]
Nov 03 08:06:32.614244 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.614217    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323b9bca-191d-4d7e-8b2b-a41ec4be28a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-598c7ad0-zqkh6\" (UID: \"323b9bca-191d-4d7e-8b2b-a41ec4be28a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6"
Nov 03 08:06:32.614376 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.614275    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qsrw\" (UniqueName: \"kubernetes.io/projected/323b9bca-191d-4d7e-8b2b-a41ec4be28a8-kube-api-access-7qsrw\") pod \"maas-api-revoke-sub-598c7ad0-zqkh6\" (UID: \"323b9bca-191d-4d7e-8b2b-a41ec4be28a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6"
Nov 03 08:06:32.614376 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.614368    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323b9bca-191d-4d7e-8b2b-a41ec4be28a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-598c7ad0-zqkh6\" (UID: \"323b9bca-191d-4d7e-8b2b-a41ec4be28a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6"
Nov 03 08:06:32.615929 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.615876    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323b9bca-191d-4d7e-8b2b-a41ec4be28a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-598c7ad0-zqkh6\" (UID: \"323b9bca-191d-4d7e-8b2b-a41ec4be28a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6"
Nov 03 08:06:32.621665 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.621636    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323b9bca-191d-4d7e-8b2b-a41ec4be28a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-598c7ad0-zqkh6\" (UID: \"323b9bca-191d-4d7e-8b2b-a41ec4be28a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6"
Nov 03 08:06:32.640952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.640926    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qsrw\" (UniqueName: \"kubernetes.io/projected/323b9bca-191d-4d7e-8b2b-a41ec4be28a8-kube-api-access-7qsrw\") pod \"maas-api-revoke-sub-598c7ad0-zqkh6\" (UID: \"323b9bca-191d-4d7e-8b2b-a41ec4be28a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6"
Nov 03 08:06:32.716127 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.716044    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cf0844f-cgb4f\" (UID: \"6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f"
Nov 03 08:06:32.716127 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.716085    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnrvz\" (UniqueName: \"kubernetes.io/projected/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2-kube-api-access-rnrvz\") pod \"maas-api-revoke-sub-6cf0844f-cgb4f\" (UID: \"6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f"
Nov 03 08:06:32.716339 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.716190    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cf0844f-cgb4f\" (UID: \"6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f"
Nov 03 08:06:32.762902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.762876    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6"
Nov 03 08:06:32.818491 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.818464    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cf0844f-cgb4f\" (UID: \"6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f"
Nov 03 08:06:32.818840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.818814    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cf0844f-cgb4f\" (UID: \"6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f"
Nov 03 08:06:32.818940 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.818860    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnrvz\" (UniqueName: \"kubernetes.io/projected/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2-kube-api-access-rnrvz\") pod \"maas-api-revoke-sub-6cf0844f-cgb4f\" (UID: \"6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f"
Nov 03 08:06:32.819604 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.819572    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6cf0844f-cgb4f\" (UID: \"6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f"
Nov 03 08:06:32.821277 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.821254    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6cf0844f-cgb4f\" (UID: \"6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f"
Nov 03 08:06:32.825458 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.825438    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnrvz\" (UniqueName: \"kubernetes.io/projected/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2-kube-api-access-rnrvz\") pod \"maas-api-revoke-sub-6cf0844f-cgb4f\" (UID: \"6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f"
Nov 03 08:06:32.919625 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:32.919591    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f"
Nov 03 08:06:33.041064 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:06:33.041038    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323b9bca_191d_4d7e_8b2b_a41ec4be28a8.slice/crio-f5720ed93a85c7c4b0f6e14ab71750d1f45fe9c22a8d2ab0a53ea755313cb07e WatchSource:0}: Error finding container f5720ed93a85c7c4b0f6e14ab71750d1f45fe9c22a8d2ab0a53ea755313cb07e: Status 404 returned error can't find the container with id f5720ed93a85c7c4b0f6e14ab71750d1f45fe9c22a8d2ab0a53ea755313cb07e
Nov 03 08:06:33.044383 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:33.044355    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6"]
Nov 03 08:06:33.204868 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:06:33.204827    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1c1a14_1bc5_4ba5_94e8_9711aa7041a2.slice/crio-7edd9d967a040c59b5395d15c7db6c8342f002d664ff928badb1a930e52e67fe WatchSource:0}: Error finding container 7edd9d967a040c59b5395d15c7db6c8342f002d664ff928badb1a930e52e67fe: Status 404 returned error can't find the container with id 7edd9d967a040c59b5395d15c7db6c8342f002d664ff928badb1a930e52e67fe
Nov 03 08:06:33.205549 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:33.205520    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f"]
Nov 03 08:06:33.716041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:33.716007    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f" event={"ID":"6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2","Type":"ContainerStarted","Data":"321f21edfafd30811b4ae4ba6d728bef69ba1a570d2ce16a1d16302bff3d8d9a"}
Nov 03 08:06:33.716520 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:33.716063    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f" event={"ID":"6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2","Type":"ContainerStarted","Data":"7edd9d967a040c59b5395d15c7db6c8342f002d664ff928badb1a930e52e67fe"}
Nov 03 08:06:33.717768 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:33.717732    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6" event={"ID":"323b9bca-191d-4d7e-8b2b-a41ec4be28a8","Type":"ContainerStarted","Data":"f4116d2eb6b57695f675f158238de954bbbad4d349b247b37d442f546e5bb49d"}
Nov 03 08:06:33.717858 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:33.717789    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6" event={"ID":"323b9bca-191d-4d7e-8b2b-a41ec4be28a8","Type":"ContainerStarted","Data":"f5720ed93a85c7c4b0f6e14ab71750d1f45fe9c22a8d2ab0a53ea755313cb07e"}
Nov 03 08:06:33.733624 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:33.733582    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f" podStartSLOduration=1.73356959 podStartE2EDuration="1.73356959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:33.72833615 +0000 UTC m=+791.842800082" watchObservedRunningTime="2026-11-03 08:06:33.73356959 +0000 UTC m=+791.848033542"
Nov 03 08:06:33.745744 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:33.745706    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6" podStartSLOduration=1.745695076 podStartE2EDuration="1.745695076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:33.742320094 +0000 UTC m=+791.856784027" watchObservedRunningTime="2026-11-03 08:06:33.745695076 +0000 UTC m=+791.860159014"
Nov 03 08:06:33.882448 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:33.882418    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dd21693-qksdt"]
Nov 03 08:06:33.887689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:33.887670    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd21693-qksdt"
Nov 03 08:06:33.895469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:33.895448    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dd21693-qksdt"]
Nov 03 08:06:33.933156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:33.933131    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66fab8d-b783-4223-8b36-edae53932323-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dd21693-qksdt\" (UID: \"f66fab8d-b783-4223-8b36-edae53932323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd21693-qksdt"
Nov 03 08:06:33.933312 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:33.933171    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhlln\" (UniqueName: \"kubernetes.io/projected/f66fab8d-b783-4223-8b36-edae53932323-kube-api-access-bhlln\") pod \"maas-api-revoke-sub-1dd21693-qksdt\" (UID: \"f66fab8d-b783-4223-8b36-edae53932323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd21693-qksdt"
Nov 03 08:06:33.933415 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:33.933393    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66fab8d-b783-4223-8b36-edae53932323-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dd21693-qksdt\" (UID: \"f66fab8d-b783-4223-8b36-edae53932323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd21693-qksdt"
Nov 03 08:06:34.034179 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.034155    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66fab8d-b783-4223-8b36-edae53932323-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dd21693-qksdt\" (UID: \"f66fab8d-b783-4223-8b36-edae53932323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd21693-qksdt"
Nov 03 08:06:34.034263 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.034191    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhlln\" (UniqueName: \"kubernetes.io/projected/f66fab8d-b783-4223-8b36-edae53932323-kube-api-access-bhlln\") pod \"maas-api-revoke-sub-1dd21693-qksdt\" (UID: \"f66fab8d-b783-4223-8b36-edae53932323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd21693-qksdt"
Nov 03 08:06:34.034328 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.034313    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66fab8d-b783-4223-8b36-edae53932323-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dd21693-qksdt\" (UID: \"f66fab8d-b783-4223-8b36-edae53932323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd21693-qksdt"
Nov 03 08:06:34.034732 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.034710    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66fab8d-b783-4223-8b36-edae53932323-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dd21693-qksdt\" (UID: \"f66fab8d-b783-4223-8b36-edae53932323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd21693-qksdt"
Nov 03 08:06:34.036462 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.036443    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66fab8d-b783-4223-8b36-edae53932323-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dd21693-qksdt\" (UID: \"f66fab8d-b783-4223-8b36-edae53932323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd21693-qksdt"
Nov 03 08:06:34.041114 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.041088    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhlln\" (UniqueName: \"kubernetes.io/projected/f66fab8d-b783-4223-8b36-edae53932323-kube-api-access-bhlln\") pod \"maas-api-revoke-sub-1dd21693-qksdt\" (UID: \"f66fab8d-b783-4223-8b36-edae53932323\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd21693-qksdt"
Nov 03 08:06:34.211433 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.211403    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd21693-qksdt"
Nov 03 08:06:34.487871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.487840    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dd21693-qksdt"]
Nov 03 08:06:34.488190 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:06:34.488153    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66fab8d_b783_4223_8b36_edae53932323.slice/crio-1f158e2aa8df4655684808b4ed5ecb1a85bffaadc36bf3abbbbf1059d5b3c382 WatchSource:0}: Error finding container 1f158e2aa8df4655684808b4ed5ecb1a85bffaadc36bf3abbbbf1059d5b3c382: Status 404 returned error can't find the container with id 1f158e2aa8df4655684808b4ed5ecb1a85bffaadc36bf3abbbbf1059d5b3c382
Nov 03 08:06:34.728111 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.728076    5070 generic.go:356] "Generic (PLEG): container finished" podID="323b9bca-191d-4d7e-8b2b-a41ec4be28a8" containerID="f4116d2eb6b57695f675f158238de954bbbad4d349b247b37d442f546e5bb49d" exitCode=0
Nov 03 08:06:34.728455 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.728159    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6" event={"ID":"323b9bca-191d-4d7e-8b2b-a41ec4be28a8","Type":"ContainerDied","Data":"f4116d2eb6b57695f675f158238de954bbbad4d349b247b37d442f546e5bb49d"}
Nov 03 08:06:34.729704 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.729677    5070 generic.go:356] "Generic (PLEG): container finished" podID="6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2" containerID="321f21edfafd30811b4ae4ba6d728bef69ba1a570d2ce16a1d16302bff3d8d9a" exitCode=0
Nov 03 08:06:34.729813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.729789    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f" event={"ID":"6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2","Type":"ContainerDied","Data":"321f21edfafd30811b4ae4ba6d728bef69ba1a570d2ce16a1d16302bff3d8d9a"}
Nov 03 08:06:34.731092 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.731070    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd21693-qksdt" event={"ID":"f66fab8d-b783-4223-8b36-edae53932323","Type":"ContainerStarted","Data":"1f158e2aa8df4655684808b4ed5ecb1a85bffaadc36bf3abbbbf1059d5b3c382"}
Nov 03 08:06:34.806729 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.806697    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1893a77-vlsl9"]
Nov 03 08:06:34.811086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.811066    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1893a77-vlsl9"
Nov 03 08:06:34.818477 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.818451    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1893a77-vlsl9"]
Nov 03 08:06:34.850649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.850626    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77bb75c9-e45c-4c45-9242-a3269c5d6549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1893a77-vlsl9\" (UID: \"77bb75c9-e45c-4c45-9242-a3269c5d6549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1893a77-vlsl9"
Nov 03 08:06:34.850926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.850900    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c427\" (UniqueName: \"kubernetes.io/projected/77bb75c9-e45c-4c45-9242-a3269c5d6549-kube-api-access-9c427\") pod \"maas-api-revoke-sub-e1893a77-vlsl9\" (UID: \"77bb75c9-e45c-4c45-9242-a3269c5d6549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1893a77-vlsl9"
Nov 03 08:06:34.851011 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.850996    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77bb75c9-e45c-4c45-9242-a3269c5d6549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1893a77-vlsl9\" (UID: \"77bb75c9-e45c-4c45-9242-a3269c5d6549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1893a77-vlsl9"
Nov 03 08:06:34.952174 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.952135    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77bb75c9-e45c-4c45-9242-a3269c5d6549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1893a77-vlsl9\" (UID: \"77bb75c9-e45c-4c45-9242-a3269c5d6549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1893a77-vlsl9"
Nov 03 08:06:34.952326 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.952273    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77bb75c9-e45c-4c45-9242-a3269c5d6549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1893a77-vlsl9\" (UID: \"77bb75c9-e45c-4c45-9242-a3269c5d6549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1893a77-vlsl9"
Nov 03 08:06:34.952456 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.952440    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c427\" (UniqueName: \"kubernetes.io/projected/77bb75c9-e45c-4c45-9242-a3269c5d6549-kube-api-access-9c427\") pod \"maas-api-revoke-sub-e1893a77-vlsl9\" (UID: \"77bb75c9-e45c-4c45-9242-a3269c5d6549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1893a77-vlsl9"
Nov 03 08:06:34.952955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.952935    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77bb75c9-e45c-4c45-9242-a3269c5d6549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1893a77-vlsl9\" (UID: \"77bb75c9-e45c-4c45-9242-a3269c5d6549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1893a77-vlsl9"
Nov 03 08:06:34.954470 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.954449    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77bb75c9-e45c-4c45-9242-a3269c5d6549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1893a77-vlsl9\" (UID: \"77bb75c9-e45c-4c45-9242-a3269c5d6549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1893a77-vlsl9"
Nov 03 08:06:34.959457 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:34.959431    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c427\" (UniqueName: \"kubernetes.io/projected/77bb75c9-e45c-4c45-9242-a3269c5d6549-kube-api-access-9c427\") pod \"maas-api-revoke-sub-e1893a77-vlsl9\" (UID: \"77bb75c9-e45c-4c45-9242-a3269c5d6549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1893a77-vlsl9"
Nov 03 08:06:35.133635 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:35.133605    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1893a77-vlsl9"
Nov 03 08:06:35.413152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:35.413073    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1893a77-vlsl9"]
Nov 03 08:06:35.427842 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:06:35.427811    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77bb75c9_e45c_4c45_9242_a3269c5d6549.slice/crio-42c8fc5c51ea75e1b3525a2189521ab43e790c690d84f58fdd2e3cb9f961fd4e WatchSource:0}: Error finding container 42c8fc5c51ea75e1b3525a2189521ab43e790c690d84f58fdd2e3cb9f961fd4e: Status 404 returned error can't find the container with id 42c8fc5c51ea75e1b3525a2189521ab43e790c690d84f58fdd2e3cb9f961fd4e
Nov 03 08:06:35.741042 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:35.741004    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1893a77-vlsl9" event={"ID":"77bb75c9-e45c-4c45-9242-a3269c5d6549","Type":"ContainerStarted","Data":"42c8fc5c51ea75e1b3525a2189521ab43e790c690d84f58fdd2e3cb9f961fd4e"}
Nov 03 08:06:35.742656 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:35.742624    5070 generic.go:356] "Generic (PLEG): container finished" podID="f66fab8d-b783-4223-8b36-edae53932323" containerID="4aade9f7ab74300bd5fce8c071ff4879b112e7fe797b90fc23ecc9e382aa808c" exitCode=0
Nov 03 08:06:35.742831 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:35.742689    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd21693-qksdt" event={"ID":"f66fab8d-b783-4223-8b36-edae53932323","Type":"ContainerDied","Data":"4aade9f7ab74300bd5fce8c071ff4879b112e7fe797b90fc23ecc9e382aa808c"}
Nov 03 08:06:35.974639 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:35.974609    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6"
Nov 03 08:06:36.070853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.070821    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/323b9bca-191d-4d7e-8b2b-a41ec4be28a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323b9bca-191d-4d7e-8b2b-a41ec4be28a8-maas-api-xks-ca\") pod \"323b9bca-191d-4d7e-8b2b-a41ec4be28a8\" (UID: \"323b9bca-191d-4d7e-8b2b-a41ec4be28a8\") "
Nov 03 08:06:36.071043 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.071023    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/323b9bca-191d-4d7e-8b2b-a41ec4be28a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323b9bca-191d-4d7e-8b2b-a41ec4be28a8-maas-api-openshift-ca\") pod \"323b9bca-191d-4d7e-8b2b-a41ec4be28a8\" (UID: \"323b9bca-191d-4d7e-8b2b-a41ec4be28a8\") "
Nov 03 08:06:36.071141 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.071123    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/323b9bca-191d-4d7e-8b2b-a41ec4be28a8-kube-api-access-7qsrw\" (UniqueName: \"kubernetes.io/projected/323b9bca-191d-4d7e-8b2b-a41ec4be28a8-kube-api-access-7qsrw\") pod \"323b9bca-191d-4d7e-8b2b-a41ec4be28a8\" (UID: \"323b9bca-191d-4d7e-8b2b-a41ec4be28a8\") "
Nov 03 08:06:36.071505 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.071470    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/323b9bca-191d-4d7e-8b2b-a41ec4be28a8-maas-api-openshift-ca" pod "323b9bca-191d-4d7e-8b2b-a41ec4be28a8" (UID: "323b9bca-191d-4d7e-8b2b-a41ec4be28a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:06:36.071796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.071775    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/323b9bca-191d-4d7e-8b2b-a41ec4be28a8-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:36.073419 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.073384    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/323b9bca-191d-4d7e-8b2b-a41ec4be28a8-maas-api-xks-ca" pod "323b9bca-191d-4d7e-8b2b-a41ec4be28a8" (UID: "323b9bca-191d-4d7e-8b2b-a41ec4be28a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:06:36.073531 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.073466    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/323b9bca-191d-4d7e-8b2b-a41ec4be28a8-kube-api-access-7qsrw" pod "323b9bca-191d-4d7e-8b2b-a41ec4be28a8" (UID: "323b9bca-191d-4d7e-8b2b-a41ec4be28a8"). InnerVolumeSpecName "kube-api-access-7qsrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:06:36.082067 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.082043    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f"
Nov 03 08:06:36.172821 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.172752    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2-maas-api-openshift-ca\") pod \"6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2\" (UID: \"6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2\") "
Nov 03 08:06:36.172965 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.172881    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2-kube-api-access-rnrvz\" (UniqueName: \"kubernetes.io/projected/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2-kube-api-access-rnrvz\") pod \"6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2\" (UID: \"6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2\") "
Nov 03 08:06:36.173028 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.173007    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2-maas-api-xks-ca\") pod \"6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2\" (UID: \"6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2\") "
Nov 03 08:06:36.173163 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.173142    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2-maas-api-openshift-ca" pod "6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2" (UID: "6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:06:36.173492 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.173467    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qsrw\" (UniqueName: \"kubernetes.io/projected/323b9bca-191d-4d7e-8b2b-a41ec4be28a8-kube-api-access-7qsrw\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:36.173492 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.173495    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/323b9bca-191d-4d7e-8b2b-a41ec4be28a8-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:36.173662 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.173509    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:36.174925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.174897    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2-maas-api-xks-ca" pod "6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2" (UID: "6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:06:36.175012 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.174918    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2-kube-api-access-rnrvz" pod "6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2" (UID: "6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2"). InnerVolumeSpecName "kube-api-access-rnrvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:06:36.275207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.275139    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnrvz\" (UniqueName: \"kubernetes.io/projected/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2-kube-api-access-rnrvz\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:36.275207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.275169    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:36.752475 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.752445    5070 generic.go:356] "Generic (PLEG): container finished" podID="77bb75c9-e45c-4c45-9242-a3269c5d6549" containerID="b5c62182f8f3b9d00b68bfc68b95b9e27e66086710dba7db67496ffd04fc19cc" exitCode=0
Nov 03 08:06:36.752887 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.752535    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1893a77-vlsl9" event={"ID":"77bb75c9-e45c-4c45-9242-a3269c5d6549","Type":"ContainerDied","Data":"b5c62182f8f3b9d00b68bfc68b95b9e27e66086710dba7db67496ffd04fc19cc"}
Nov 03 08:06:36.754021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.754000    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6"
Nov 03 08:06:36.754142 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.754043    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6" event={"ID":"323b9bca-191d-4d7e-8b2b-a41ec4be28a8","Type":"ContainerDied","Data":"f5720ed93a85c7c4b0f6e14ab71750d1f45fe9c22a8d2ab0a53ea755313cb07e"}
Nov 03 08:06:36.754142 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.754084    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5720ed93a85c7c4b0f6e14ab71750d1f45fe9c22a8d2ab0a53ea755313cb07e"
Nov 03 08:06:36.755418 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.755396    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f"
Nov 03 08:06:36.755532 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.755507    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f" event={"ID":"6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2","Type":"ContainerDied","Data":"7edd9d967a040c59b5395d15c7db6c8342f002d664ff928badb1a930e52e67fe"}
Nov 03 08:06:36.755655 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.755552    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7edd9d967a040c59b5395d15c7db6c8342f002d664ff928badb1a930e52e67fe"
Nov 03 08:06:36.947508 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:36.947485    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd21693-qksdt"
Nov 03 08:06:37.086570 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:37.086538    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f66fab8d-b783-4223-8b36-edae53932323-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66fab8d-b783-4223-8b36-edae53932323-maas-api-openshift-ca\") pod \"f66fab8d-b783-4223-8b36-edae53932323\" (UID: \"f66fab8d-b783-4223-8b36-edae53932323\") "
Nov 03 08:06:37.086738 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:37.086723    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f66fab8d-b783-4223-8b36-edae53932323-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66fab8d-b783-4223-8b36-edae53932323-maas-api-xks-ca\") pod \"f66fab8d-b783-4223-8b36-edae53932323\" (UID: \"f66fab8d-b783-4223-8b36-edae53932323\") "
Nov 03 08:06:37.086833 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:37.086809    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f66fab8d-b783-4223-8b36-edae53932323-kube-api-access-bhlln\" (UniqueName: \"kubernetes.io/projected/f66fab8d-b783-4223-8b36-edae53932323-kube-api-access-bhlln\") pod \"f66fab8d-b783-4223-8b36-edae53932323\" (UID: \"f66fab8d-b783-4223-8b36-edae53932323\") "
Nov 03 08:06:37.086990 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:37.086965    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f66fab8d-b783-4223-8b36-edae53932323-maas-api-openshift-ca" pod "f66fab8d-b783-4223-8b36-edae53932323" (UID: "f66fab8d-b783-4223-8b36-edae53932323"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:06:37.087350 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:37.087326    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f66fab8d-b783-4223-8b36-edae53932323-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:37.089223 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:37.089196    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f66fab8d-b783-4223-8b36-edae53932323-kube-api-access-bhlln" pod "f66fab8d-b783-4223-8b36-edae53932323" (UID: "f66fab8d-b783-4223-8b36-edae53932323"). InnerVolumeSpecName "kube-api-access-bhlln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:06:37.089386 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:37.089363    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f66fab8d-b783-4223-8b36-edae53932323-maas-api-xks-ca" pod "f66fab8d-b783-4223-8b36-edae53932323" (UID: "f66fab8d-b783-4223-8b36-edae53932323"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:06:37.189077 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:37.189022    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f66fab8d-b783-4223-8b36-edae53932323-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:37.189077 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:37.189047    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhlln\" (UniqueName: \"kubernetes.io/projected/f66fab8d-b783-4223-8b36-edae53932323-kube-api-access-bhlln\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:37.764061 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:37.764033    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd21693-qksdt"
Nov 03 08:06:37.764480 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:37.764056    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dd21693-qksdt" event={"ID":"f66fab8d-b783-4223-8b36-edae53932323","Type":"ContainerDied","Data":"1f158e2aa8df4655684808b4ed5ecb1a85bffaadc36bf3abbbbf1059d5b3c382"}
Nov 03 08:06:37.764480 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:37.764119    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f158e2aa8df4655684808b4ed5ecb1a85bffaadc36bf3abbbbf1059d5b3c382"
Nov 03 08:06:37.944607 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:37.944582    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1893a77-vlsl9"
Nov 03 08:06:38.100464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:38.100437    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77bb75c9-e45c-4c45-9242-a3269c5d6549-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77bb75c9-e45c-4c45-9242-a3269c5d6549-maas-api-openshift-ca\") pod \"77bb75c9-e45c-4c45-9242-a3269c5d6549\" (UID: \"77bb75c9-e45c-4c45-9242-a3269c5d6549\") "
Nov 03 08:06:38.100726 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:38.100517    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77bb75c9-e45c-4c45-9242-a3269c5d6549-kube-api-access-9c427\" (UniqueName: \"kubernetes.io/projected/77bb75c9-e45c-4c45-9242-a3269c5d6549-kube-api-access-9c427\") pod \"77bb75c9-e45c-4c45-9242-a3269c5d6549\" (UID: \"77bb75c9-e45c-4c45-9242-a3269c5d6549\") "
Nov 03 08:06:38.100726 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:38.100568    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77bb75c9-e45c-4c45-9242-a3269c5d6549-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77bb75c9-e45c-4c45-9242-a3269c5d6549-maas-api-xks-ca\") pod \"77bb75c9-e45c-4c45-9242-a3269c5d6549\" (UID: \"77bb75c9-e45c-4c45-9242-a3269c5d6549\") "
Nov 03 08:06:38.100877 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:38.100803    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77bb75c9-e45c-4c45-9242-a3269c5d6549-maas-api-openshift-ca" pod "77bb75c9-e45c-4c45-9242-a3269c5d6549" (UID: "77bb75c9-e45c-4c45-9242-a3269c5d6549"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:06:38.100996 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:38.100977    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77bb75c9-e45c-4c45-9242-a3269c5d6549-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:38.102801 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:38.102746    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77bb75c9-e45c-4c45-9242-a3269c5d6549-maas-api-xks-ca" pod "77bb75c9-e45c-4c45-9242-a3269c5d6549" (UID: "77bb75c9-e45c-4c45-9242-a3269c5d6549"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:06:38.102938 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:38.102903    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77bb75c9-e45c-4c45-9242-a3269c5d6549-kube-api-access-9c427" pod "77bb75c9-e45c-4c45-9242-a3269c5d6549" (UID: "77bb75c9-e45c-4c45-9242-a3269c5d6549"). InnerVolumeSpecName "kube-api-access-9c427". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:06:38.201751 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:38.201728    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c427\" (UniqueName: \"kubernetes.io/projected/77bb75c9-e45c-4c45-9242-a3269c5d6549-kube-api-access-9c427\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:38.201751 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:38.201751    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77bb75c9-e45c-4c45-9242-a3269c5d6549-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:38.773907 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:38.773878    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1893a77-vlsl9"
Nov 03 08:06:38.774278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:38.773920    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1893a77-vlsl9" event={"ID":"77bb75c9-e45c-4c45-9242-a3269c5d6549","Type":"ContainerDied","Data":"42c8fc5c51ea75e1b3525a2189521ab43e790c690d84f58fdd2e3cb9f961fd4e"}
Nov 03 08:06:38.774278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:38.773971    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42c8fc5c51ea75e1b3525a2189521ab43e790c690d84f58fdd2e3cb9f961fd4e"
Nov 03 08:06:41.177280 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.177246    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58ce4c6e-xhg94"]
Nov 03 08:06:41.177902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.177887    5070 cpu_manager.go:414] "containerMap: removing container" podUID="6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2" containerName="revoke-keys"
Nov 03 08:06:41.177957 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.177903    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2" containerName="revoke-keys"
Nov 03 08:06:41.177957 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.177930    5070 cpu_manager.go:414] "containerMap: removing container" podUID="77bb75c9-e45c-4c45-9242-a3269c5d6549" containerName="revoke-keys"
Nov 03 08:06:41.177957 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.177935    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77bb75c9-e45c-4c45-9242-a3269c5d6549" containerName="revoke-keys"
Nov 03 08:06:41.177957 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.177950    5070 cpu_manager.go:414] "containerMap: removing container" podUID="323b9bca-191d-4d7e-8b2b-a41ec4be28a8" containerName="revoke-keys"
Nov 03 08:06:41.177957 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.177955    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="323b9bca-191d-4d7e-8b2b-a41ec4be28a8" containerName="revoke-keys"
Nov 03 08:06:41.178155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.177969    5070 cpu_manager.go:414] "containerMap: removing container" podUID="f66fab8d-b783-4223-8b36-edae53932323" containerName="revoke-keys"
Nov 03 08:06:41.178155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.177975    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f66fab8d-b783-4223-8b36-edae53932323" containerName="revoke-keys"
Nov 03 08:06:41.178155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.178074    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="77bb75c9-e45c-4c45-9242-a3269c5d6549" containerName="revoke-keys"
Nov 03 08:06:41.178155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.178084    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="323b9bca-191d-4d7e-8b2b-a41ec4be28a8" containerName="revoke-keys"
Nov 03 08:06:41.178155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.178093    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="f66fab8d-b783-4223-8b36-edae53932323" containerName="revoke-keys"
Nov 03 08:06:41.178155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.178103    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2" containerName="revoke-keys"
Nov 03 08:06:41.182141 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.182124    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ce4c6e-xhg94"
Nov 03 08:06:41.185168 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.185145    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:06:41.185286 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.185205    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:06:41.185286 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.185265    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:06:41.185789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.185737    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:06:41.189698 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.189676    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58ce4c6e-xhg94"]
Nov 03 08:06:41.342661 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.342634    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87n97\" (UniqueName: \"kubernetes.io/projected/1703c4f9-5aea-4c19-8d85-d9040bc83baa-kube-api-access-87n97\") pod \"maas-api-revoke-sub-58ce4c6e-xhg94\" (UID: \"1703c4f9-5aea-4c19-8d85-d9040bc83baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ce4c6e-xhg94"
Nov 03 08:06:41.342833 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.342809    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1703c4f9-5aea-4c19-8d85-d9040bc83baa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58ce4c6e-xhg94\" (UID: \"1703c4f9-5aea-4c19-8d85-d9040bc83baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ce4c6e-xhg94"
Nov 03 08:06:41.342892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.342867    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1703c4f9-5aea-4c19-8d85-d9040bc83baa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58ce4c6e-xhg94\" (UID: \"1703c4f9-5aea-4c19-8d85-d9040bc83baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ce4c6e-xhg94"
Nov 03 08:06:41.443724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.443657    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87n97\" (UniqueName: \"kubernetes.io/projected/1703c4f9-5aea-4c19-8d85-d9040bc83baa-kube-api-access-87n97\") pod \"maas-api-revoke-sub-58ce4c6e-xhg94\" (UID: \"1703c4f9-5aea-4c19-8d85-d9040bc83baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ce4c6e-xhg94"
Nov 03 08:06:41.443833 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.443774    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1703c4f9-5aea-4c19-8d85-d9040bc83baa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58ce4c6e-xhg94\" (UID: \"1703c4f9-5aea-4c19-8d85-d9040bc83baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ce4c6e-xhg94"
Nov 03 08:06:41.443833 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.443825    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1703c4f9-5aea-4c19-8d85-d9040bc83baa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58ce4c6e-xhg94\" (UID: \"1703c4f9-5aea-4c19-8d85-d9040bc83baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ce4c6e-xhg94"
Nov 03 08:06:41.444529 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.444509    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1703c4f9-5aea-4c19-8d85-d9040bc83baa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58ce4c6e-xhg94\" (UID: \"1703c4f9-5aea-4c19-8d85-d9040bc83baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ce4c6e-xhg94"
Nov 03 08:06:41.446225 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.446205    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1703c4f9-5aea-4c19-8d85-d9040bc83baa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58ce4c6e-xhg94\" (UID: \"1703c4f9-5aea-4c19-8d85-d9040bc83baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ce4c6e-xhg94"
Nov 03 08:06:41.450359 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.450339    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87n97\" (UniqueName: \"kubernetes.io/projected/1703c4f9-5aea-4c19-8d85-d9040bc83baa-kube-api-access-87n97\") pod \"maas-api-revoke-sub-58ce4c6e-xhg94\" (UID: \"1703c4f9-5aea-4c19-8d85-d9040bc83baa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ce4c6e-xhg94"
Nov 03 08:06:41.509460 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.509426    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ce4c6e-xhg94"
Nov 03 08:06:41.801492 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:06:41.801447    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1703c4f9_5aea_4c19_8d85_d9040bc83baa.slice/crio-c2902e7457ad05a7b64e54d653f3829d6adf127762555a7b6865a457a1fe0386 WatchSource:0}: Error finding container c2902e7457ad05a7b64e54d653f3829d6adf127762555a7b6865a457a1fe0386: Status 404 returned error can't find the container with id c2902e7457ad05a7b64e54d653f3829d6adf127762555a7b6865a457a1fe0386
Nov 03 08:06:41.801632 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:41.801609    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58ce4c6e-xhg94"]
Nov 03 08:06:42.805489 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:42.805460    5070 generic.go:356] "Generic (PLEG): container finished" podID="1703c4f9-5aea-4c19-8d85-d9040bc83baa" containerID="42239431bc9311b7c1ad6a351832007fafa7de84172bd8f45f8322498c56a319" exitCode=0
Nov 03 08:06:42.805829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:42.805513    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ce4c6e-xhg94" event={"ID":"1703c4f9-5aea-4c19-8d85-d9040bc83baa","Type":"ContainerDied","Data":"42239431bc9311b7c1ad6a351832007fafa7de84172bd8f45f8322498c56a319"}
Nov 03 08:06:42.805829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:42.805555    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ce4c6e-xhg94" event={"ID":"1703c4f9-5aea-4c19-8d85-d9040bc83baa","Type":"ContainerStarted","Data":"c2902e7457ad05a7b64e54d653f3829d6adf127762555a7b6865a457a1fe0386"}
Nov 03 08:06:44.022423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:44.022401    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ce4c6e-xhg94"
Nov 03 08:06:44.188138 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:44.188110    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1703c4f9-5aea-4c19-8d85-d9040bc83baa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1703c4f9-5aea-4c19-8d85-d9040bc83baa-maas-api-xks-ca\") pod \"1703c4f9-5aea-4c19-8d85-d9040bc83baa\" (UID: \"1703c4f9-5aea-4c19-8d85-d9040bc83baa\") "
Nov 03 08:06:44.188321 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:44.188175    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1703c4f9-5aea-4c19-8d85-d9040bc83baa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1703c4f9-5aea-4c19-8d85-d9040bc83baa-maas-api-openshift-ca\") pod \"1703c4f9-5aea-4c19-8d85-d9040bc83baa\" (UID: \"1703c4f9-5aea-4c19-8d85-d9040bc83baa\") "
Nov 03 08:06:44.188321 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:44.188275    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1703c4f9-5aea-4c19-8d85-d9040bc83baa-kube-api-access-87n97\" (UniqueName: \"kubernetes.io/projected/1703c4f9-5aea-4c19-8d85-d9040bc83baa-kube-api-access-87n97\") pod \"1703c4f9-5aea-4c19-8d85-d9040bc83baa\" (UID: \"1703c4f9-5aea-4c19-8d85-d9040bc83baa\") "
Nov 03 08:06:44.188524 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:44.188504    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1703c4f9-5aea-4c19-8d85-d9040bc83baa-maas-api-openshift-ca" pod "1703c4f9-5aea-4c19-8d85-d9040bc83baa" (UID: "1703c4f9-5aea-4c19-8d85-d9040bc83baa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:06:44.188893 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:44.188864    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1703c4f9-5aea-4c19-8d85-d9040bc83baa-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:44.190316 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:44.190288    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1703c4f9-5aea-4c19-8d85-d9040bc83baa-maas-api-xks-ca" pod "1703c4f9-5aea-4c19-8d85-d9040bc83baa" (UID: "1703c4f9-5aea-4c19-8d85-d9040bc83baa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:06:44.190580 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:44.190552    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1703c4f9-5aea-4c19-8d85-d9040bc83baa-kube-api-access-87n97" pod "1703c4f9-5aea-4c19-8d85-d9040bc83baa" (UID: "1703c4f9-5aea-4c19-8d85-d9040bc83baa"). InnerVolumeSpecName "kube-api-access-87n97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:06:44.290074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:44.290045    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1703c4f9-5aea-4c19-8d85-d9040bc83baa-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:44.290074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:44.290073    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87n97\" (UniqueName: \"kubernetes.io/projected/1703c4f9-5aea-4c19-8d85-d9040bc83baa-kube-api-access-87n97\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:44.822655 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:44.822626    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ce4c6e-xhg94"
Nov 03 08:06:44.822886 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:44.822621    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58ce4c6e-xhg94" event={"ID":"1703c4f9-5aea-4c19-8d85-d9040bc83baa","Type":"ContainerDied","Data":"c2902e7457ad05a7b64e54d653f3829d6adf127762555a7b6865a457a1fe0386"}
Nov 03 08:06:44.822886 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:44.822789    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2902e7457ad05a7b64e54d653f3829d6adf127762555a7b6865a457a1fe0386"
Nov 03 08:06:45.584181 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.584150    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6"]
Nov 03 08:06:45.585022 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.585001    5070 cpu_manager.go:414] "containerMap: removing container" podUID="1703c4f9-5aea-4c19-8d85-d9040bc83baa" containerName="revoke-keys"
Nov 03 08:06:45.585022 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.585022    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1703c4f9-5aea-4c19-8d85-d9040bc83baa" containerName="revoke-keys"
Nov 03 08:06:45.585219 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.585205    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="1703c4f9-5aea-4c19-8d85-d9040bc83baa" containerName="revoke-keys"
Nov 03 08:06:45.589129 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.589104    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6"
Nov 03 08:06:45.591943 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.591920    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:06:45.592071 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.591931    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:06:45.592146 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.592068    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:06:45.592250 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.592235    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:06:45.595647 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.595622    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6"]
Nov 03 08:06:45.611803 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.611770    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48af7f66-65e9-4ef0-b99a-7046e3303bef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac62b195-kz2j6\" (UID: \"48af7f66-65e9-4ef0-b99a-7046e3303bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6"
Nov 03 08:06:45.612161 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.612140    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5g95\" (UniqueName: \"kubernetes.io/projected/48af7f66-65e9-4ef0-b99a-7046e3303bef-kube-api-access-b5g95\") pod \"maas-api-revoke-sub-ac62b195-kz2j6\" (UID: \"48af7f66-65e9-4ef0-b99a-7046e3303bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6"
Nov 03 08:06:45.612295 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.612279    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48af7f66-65e9-4ef0-b99a-7046e3303bef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac62b195-kz2j6\" (UID: \"48af7f66-65e9-4ef0-b99a-7046e3303bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6"
Nov 03 08:06:45.714114 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.714084    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48af7f66-65e9-4ef0-b99a-7046e3303bef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac62b195-kz2j6\" (UID: \"48af7f66-65e9-4ef0-b99a-7046e3303bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6"
Nov 03 08:06:45.714241 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.714199    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5g95\" (UniqueName: \"kubernetes.io/projected/48af7f66-65e9-4ef0-b99a-7046e3303bef-kube-api-access-b5g95\") pod \"maas-api-revoke-sub-ac62b195-kz2j6\" (UID: \"48af7f66-65e9-4ef0-b99a-7046e3303bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6"
Nov 03 08:06:45.714241 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.714219    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48af7f66-65e9-4ef0-b99a-7046e3303bef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac62b195-kz2j6\" (UID: \"48af7f66-65e9-4ef0-b99a-7046e3303bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6"
Nov 03 08:06:45.714881 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.714857    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48af7f66-65e9-4ef0-b99a-7046e3303bef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac62b195-kz2j6\" (UID: \"48af7f66-65e9-4ef0-b99a-7046e3303bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6"
Nov 03 08:06:45.716551 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.716526    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48af7f66-65e9-4ef0-b99a-7046e3303bef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac62b195-kz2j6\" (UID: \"48af7f66-65e9-4ef0-b99a-7046e3303bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6"
Nov 03 08:06:45.721403 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.721380    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5g95\" (UniqueName: \"kubernetes.io/projected/48af7f66-65e9-4ef0-b99a-7046e3303bef-kube-api-access-b5g95\") pod \"maas-api-revoke-sub-ac62b195-kz2j6\" (UID: \"48af7f66-65e9-4ef0-b99a-7046e3303bef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6"
Nov 03 08:06:45.762739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.762715    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"]
Nov 03 08:06:45.767453 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.767434    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.776644 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.776617    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"]
Nov 03 08:06:45.781711 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.781694    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"]
Nov 03 08:06:45.781842 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.781795    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.811700 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.811667    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"]
Nov 03 08:06:45.815391 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.815362    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"]
Nov 03 08:06:45.815733 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.815707    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0822706-f191-47c7-8c87-e373872590e3-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.815828 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.815804    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.815892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.815837    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.815892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.815863    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.816002 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.815891    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.816002 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.815941    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.816002 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.815971    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.816163 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.815999    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.816163 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.816031    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsx2t\" (UniqueName: \"kubernetes.io/projected/c0822706-f191-47c7-8c87-e373872590e3-kube-api-access-fsx2t\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.816163 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.816101    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.816163 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.816127    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.816360 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.816167    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e8f246-f7e4-4dfa-afef-0fb361672461-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.816360 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.816224    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0822706-f191-47c7-8c87-e373872590e3-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.816360 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.816265    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0822706-f191-47c7-8c87-e373872590e3-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.816360 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.816291    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.816562 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.816405    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.816562 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.816438    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.816562 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.816484    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wszhb\" (UniqueName: \"kubernetes.io/projected/17e8f246-f7e4-4dfa-afef-0fb361672461-kube-api-access-wszhb\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.822626 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.822600    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:45.827841 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.827818    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"]
Nov 03 08:06:45.844329 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.844303    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"]
Nov 03 08:06:45.849798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.849749    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:45.879434 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.879407    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"]
Nov 03 08:06:45.907539 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.907506    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6"
Nov 03 08:06:45.917854 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.917828    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:45.917954 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.917873    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:45.918024 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.917990    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.918092 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.918029    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.918092 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.918057    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:45.918202 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.918091    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:45.918202 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.918119    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:45.918202 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.918153    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wszhb\" (UniqueName: \"kubernetes.io/projected/17e8f246-f7e4-4dfa-afef-0fb361672461-kube-api-access-wszhb\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.918202 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.918197    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:45.918415 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.918236    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4tx5\" (UniqueName: \"kubernetes.io/projected/df4cc63e-36c7-4e6e-87ab-64dea6237e42-kube-api-access-s4tx5\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:45.918415 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.918280    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:45.918415 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.918311    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0822706-f191-47c7-8c87-e373872590e3-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.918415 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.918356    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.918415 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.918384    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.918415 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.918408    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.918846 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.918406    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.918846 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.918675    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.919111 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.919085    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.919265 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.919239    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.919371 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.919284    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.919521 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.919498    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:45.919591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.919565    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:45.919649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.919612    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:45.919811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.919791    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.919906 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.919843    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.919906 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.919876    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.920022 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.919909    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsx2t\" (UniqueName: \"kubernetes.io/projected/c0822706-f191-47c7-8c87-e373872590e3-kube-api-access-fsx2t\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.920022 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.919981    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.920022 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.920010    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.920179 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.920040    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgwr2\" (UniqueName: \"kubernetes.io/projected/64c976ca-8c6d-4835-b5ca-ba88be627ea7-kube-api-access-rgwr2\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:45.920179 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.920074    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:45.920179 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.920109    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e8f246-f7e4-4dfa-afef-0fb361672461-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.920179 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.920163    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0822706-f191-47c7-8c87-e373872590e3-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.920404 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.920207    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0822706-f191-47c7-8c87-e373872590e3-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.920404 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.920233    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.920404 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.920294    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:45.920404 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.920321    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:45.920404 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.920358    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:45.920404 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.920388    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:45.920721 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.920415    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:45.921461 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.921434    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e8f246-f7e4-4dfa-afef-0fb361672461-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.921719 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.921706    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-workload-certs\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.921960 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.921941    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-credential-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.922942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.922772    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-data\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.922942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.922885    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-workload-socket\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.923101 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.922974    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.923304 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.923284    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.923596 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.923571    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0822706-f191-47c7-8c87-e373872590e3-istiod-ca-cert\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.924123 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.924101    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0822706-f191-47c7-8c87-e373872590e3-istio-podinfo\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.924970 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.924942    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-envoy\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.941072 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.941050    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0822706-f191-47c7-8c87-e373872590e3-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.943914 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.943889    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-token\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.944054 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.944030    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wszhb\" (UniqueName: \"kubernetes.io/projected/17e8f246-f7e4-4dfa-afef-0fb361672461-kube-api-access-wszhb\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:45.944421 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.944405    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsx2t\" (UniqueName: \"kubernetes.io/projected/c0822706-f191-47c7-8c87-e373872590e3-kube-api-access-fsx2t\") pod \"e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") " pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:45.972622 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.972590    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"]
Nov 03 08:06:45.981836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:45.980979    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.025745    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.025803    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.025829    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.025869    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12307292-4369-4f20-9288-9b2005609416-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.025934    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qkxr\" (UniqueName: \"kubernetes.io/projected/12307292-4369-4f20-9288-9b2005609416-kube-api-access-2qkxr\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.025972    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.026026    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgwr2\" (UniqueName: \"kubernetes.io/projected/64c976ca-8c6d-4835-b5ca-ba88be627ea7-kube-api-access-rgwr2\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.026057    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.026117    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12307292-4369-4f20-9288-9b2005609416-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.026184    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.026219    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.026247    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.026278    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.026304    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.026332    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.026358    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.026390    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.026416    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12307292-4369-4f20-9288-9b2005609416-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.026460    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.026509    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.026546    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.026573    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.026628    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.026663    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4tx5\" (UniqueName: \"kubernetes.io/projected/df4cc63e-36c7-4e6e-87ab-64dea6237e42-kube-api-access-s4tx5\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.026710    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:46.026781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.026742    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.028219 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.027322    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:46.029833 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.028800    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:46.029833 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.028920    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-workload-certs\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:46.029833 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.029438    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:46.030783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.030480    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:46.030783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.030691    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:46.031528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.031146    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:46.031528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.031249    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.034001 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.033969    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istiod-ca-cert\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:46.034685 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.034457    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-credential-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:46.034685 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.034582    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-data\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:46.034685 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.034621    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:46.034685 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.034646    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-workload-socket\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:46.034972 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.034948    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-podinfo\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:46.035923 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.035905    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-envoy\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:46.036712 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.036691    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"]
Nov 03 08:06:46.040192 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.040167    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"]
Nov 03 08:06:46.046233 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.046217    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.057194 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.056801    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4tx5\" (UniqueName: \"kubernetes.io/projected/df4cc63e-36c7-4e6e-87ab-64dea6237e42-kube-api-access-s4tx5\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:46.057194 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.057152    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:46.057491 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.057466    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgwr2\" (UniqueName: \"kubernetes.io/projected/64c976ca-8c6d-4835-b5ca-ba88be627ea7-kube-api-access-rgwr2\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:46.059827 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.059783    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-token\") pod \"e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") " pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:46.071252 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.071229    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"]
Nov 03 08:06:46.085261 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.085202    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:46.104288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.104267    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:46.132574 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.132540    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.132728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.132606    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12307292-4369-4f20-9288-9b2005609416-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.132728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.132638    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.132728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.132663    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljjfs\" (UniqueName: \"kubernetes.io/projected/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-kube-api-access-ljjfs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.132728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.132724    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.133003 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.132800    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.133003 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.132847    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.133003 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.132902    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12307292-4369-4f20-9288-9b2005609416-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.133003 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.132945    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.133199 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.133019    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.133546 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.133334    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.133546 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.133376    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.133546 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.133411    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.133546 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.133479    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-workload-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.133546 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.133487    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.133890 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.133685    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-istio-data\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.133890 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.133774    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12307292-4369-4f20-9288-9b2005609416-istiod-ca-cert\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.133890 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.133798    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12307292-4369-4f20-9288-9b2005609416-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.134044 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.133985    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qkxr\" (UniqueName: \"kubernetes.io/projected/12307292-4369-4f20-9288-9b2005609416-kube-api-access-2qkxr\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.134044 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.134009    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-workload-certs\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.134160 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.134059    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.134160 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.134105    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.134406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.134383    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-credential-socket\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.135250 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.135229    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-istio-envoy\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.136865 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.136823    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12307292-4369-4f20-9288-9b2005609416-istio-podinfo\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.140513 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.140495    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qkxr\" (UniqueName: \"kubernetes.io/projected/12307292-4369-4f20-9288-9b2005609416-kube-api-access-2qkxr\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.141752 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.141732    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12307292-4369-4f20-9288-9b2005609416-istio-token\") pod \"e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") " pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.150202 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.148729    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:46.169798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.168419    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:46.221996 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.221910    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6"]
Nov 03 08:06:46.235502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.235094    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.235502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.235150    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.235502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.235178    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.235502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.235194    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljjfs\" (UniqueName: \"kubernetes.io/projected/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-kube-api-access-ljjfs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.235502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.235235    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.235502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.235259    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.235502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.235320    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.235502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.235371    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.235870 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.235402    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.236341 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.236320    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.237165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.237145    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.238010 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.237988    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.238947 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.238925    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.239341 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.239298    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.240809 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.240782    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.242324 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.242285    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.242916 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.242875    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.244671 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.244649    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljjfs\" (UniqueName: \"kubernetes.io/projected/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-kube-api-access-ljjfs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.301060 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.301027    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:46.366699 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.366604    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:46.392623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.392591    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"]
Nov 03 08:06:46.397648 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:06:46.397618    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0822706_f191_47c7_8c87_e373872590e3.slice/crio-a42525886e4fb381e03d003883a5074d19b6840aa9727cd623878f3683a25563 WatchSource:0}: Error finding container a42525886e4fb381e03d003883a5074d19b6840aa9727cd623878f3683a25563: Status 404 returned error can't find the container with id a42525886e4fb381e03d003883a5074d19b6840aa9727cd623878f3683a25563
Nov 03 08:06:46.402251 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.402213    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:46.402363 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.402278    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:46.402363 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.402307    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:46.414334 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.413748    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"]
Nov 03 08:06:46.450252 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.450203    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:46.450502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.450475    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:46.450679 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.450661    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:46.489775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.489737    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"]
Nov 03 08:06:46.494373 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.493523    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:46.494373 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.493621    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:46.494373 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.493672    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:46.506858 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.506802    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"]
Nov 03 08:06:46.521263 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.521227    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:46.521472 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.521449    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:46.521637 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.521618    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:46.693450 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.686428    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"]
Nov 03 08:06:46.699685 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.699644    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:46.699955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.699932    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:46.700072 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.700058    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:46.729827 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.729795    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"]
Nov 03 08:06:46.751560 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.751521    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:46.751673 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.751636    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:46.751742 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.751714    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:06:46.849679 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.849648    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" event={"ID":"17e8f246-f7e4-4dfa-afef-0fb361672461","Type":"ContainerStarted","Data":"1fbbaf459fb90ee32385f9047f532e8ad7c2e1957187ff3c870517e885d4090a"}
Nov 03 08:06:46.849873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.849709    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" event={"ID":"17e8f246-f7e4-4dfa-afef-0fb361672461","Type":"ContainerStarted","Data":"a0377f95201a4ae5a147443f9c4adb873197980fa47a2855efcb47ceba51b406"}
Nov 03 08:06:46.851773 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.851683    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6" event={"ID":"48af7f66-65e9-4ef0-b99a-7046e3303bef","Type":"ContainerStarted","Data":"8d8115bb7c8959d8ef3c3ac545d5d8542e4797b275f0a0314e1dbdd33791404b"}
Nov 03 08:06:46.851773 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.851736    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6" event={"ID":"48af7f66-65e9-4ef0-b99a-7046e3303bef","Type":"ContainerStarted","Data":"0155fcb56f781150687d000a84f9351a4578e7abe4180cc8138466aada451f6a"}
Nov 03 08:06:46.854999 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.854973    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5" event={"ID":"d1a356ec-fba8-45e1-87ba-13950f3e1fcb","Type":"ContainerStarted","Data":"ffd2bad394dae91aea11bdff6265f773dee642327324ca96bc1f77361e713a5c"}
Nov 03 08:06:46.856662 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.856638    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2" event={"ID":"12307292-4369-4f20-9288-9b2005609416","Type":"ContainerStarted","Data":"e18097d08266a5336bcecf5038f162a160fffab4f7e444d44fbc079972501a0e"}
Nov 03 08:06:46.860819 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.860109    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" event={"ID":"df4cc63e-36c7-4e6e-87ab-64dea6237e42","Type":"ContainerStarted","Data":"51d9e728d05eb9ccc97d671ec61308aea3135db9f7491831c67acbf0409a301e"}
Nov 03 08:06:46.860819 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.860155    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" event={"ID":"df4cc63e-36c7-4e6e-87ab-64dea6237e42","Type":"ContainerStarted","Data":"dbd37e560bad7f669186c031613fcda1c37396481ba191eceedc3f43534af13a"}
Nov 03 08:06:46.864960 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.864275    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" event={"ID":"c0822706-f191-47c7-8c87-e373872590e3","Type":"ContainerStarted","Data":"852a632cb1812b8fba2b1ae708c51a02a9a8a0f2fe587cded79faafd9309df23"}
Nov 03 08:06:46.864960 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.864328    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" event={"ID":"c0822706-f191-47c7-8c87-e373872590e3","Type":"ContainerStarted","Data":"a42525886e4fb381e03d003883a5074d19b6840aa9727cd623878f3683a25563"}
Nov 03 08:06:46.870479 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.870434    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" podStartSLOduration=1.870417261 podStartE2EDuration="1.870417261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:46.866933568 +0000 UTC m=+804.981397511" watchObservedRunningTime="2026-11-03 08:06:46.870417261 +0000 UTC m=+804.984881193"
Nov 03 08:06:46.873342 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.872695    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr" event={"ID":"64c976ca-8c6d-4835-b5ca-ba88be627ea7","Type":"ContainerStarted","Data":"858436e67cd13dbb219a36a6c0c6514844e5c2e2eda96b78203c916c230fc665"}
Nov 03 08:06:46.873342 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.872741    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr" event={"ID":"64c976ca-8c6d-4835-b5ca-ba88be627ea7","Type":"ContainerStarted","Data":"f714c87a8f8b50f7d562ea321ca965310fea49bc81863a9d0b97116b1ea2afdb"}
Nov 03 08:06:46.894220 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.890733    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" podStartSLOduration=1.89071696 podStartE2EDuration="1.89071696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:46.88670593 +0000 UTC m=+805.001169853" watchObservedRunningTime="2026-11-03 08:06:46.89071696 +0000 UTC m=+805.005180892"
Nov 03 08:06:46.910937 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.910887    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podStartSLOduration=1.910868233 podStartE2EDuration="1.910868233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:46.905514479 +0000 UTC m=+805.019978434" watchObservedRunningTime="2026-11-03 08:06:46.910868233 +0000 UTC m=+805.025332171"
Nov 03 08:06:46.940172 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.940109    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6" podStartSLOduration=1.940089107 podStartE2EDuration="1.940089107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:46.935561037 +0000 UTC m=+805.050024969" watchObservedRunningTime="2026-11-03 08:06:46.940089107 +0000 UTC m=+805.054553039"
Nov 03 08:06:46.961514 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:46.961463    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr" podStartSLOduration=1.961449716 podStartE2EDuration="1.961449716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:46.957098868 +0000 UTC m=+805.071562799" watchObservedRunningTime="2026-11-03 08:06:46.961449716 +0000 UTC m=+805.075913647"
Nov 03 08:06:47.108588 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:47.107262    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:06:47.109025 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:47.108999    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:06:47.119840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:47.113293    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 08:06:47.127952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:47.127915    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 08:06:47.128415 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:47.124902    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 08:06:47.128515 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:47.128435    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 08:06:47.162103 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:47.151249    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:06:47.171359 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:47.162883    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 08:06:47.171359 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:47.162938    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr" podUID="64c976ca-8c6d-4835-b5ca-ba88be627ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 08:06:47.171359 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:47.169813    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:06:47.171710 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:47.171693    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 08:06:47.171859 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:47.171835    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 08:06:48.009785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:47.989154    5070 generic.go:356] "Generic (PLEG): container finished" podID="48af7f66-65e9-4ef0-b99a-7046e3303bef" containerID="8d8115bb7c8959d8ef3c3ac545d5d8542e4797b275f0a0314e1dbdd33791404b" exitCode=0
Nov 03 08:06:48.011156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.010353    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6" event={"ID":"48af7f66-65e9-4ef0-b99a-7046e3303bef","Type":"ContainerDied","Data":"8d8115bb7c8959d8ef3c3ac545d5d8542e4797b275f0a0314e1dbdd33791404b"}
Nov 03 08:06:48.167204 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.098867    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 08:06:48.168457 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.168405    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 08:06:48.180010 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.170825    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 08:06:48.180184 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.157888    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 08:06:48.180306 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.180280    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr" podUID="64c976ca-8c6d-4835-b5ca-ba88be627ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 08:06:48.180459 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.180442    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 08:06:48.201884 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.201846    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 08:06:48.202043 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.201920    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 08:06:48.244008 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.243971    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5" event={"ID":"d1a356ec-fba8-45e1-87ba-13950f3e1fcb","Type":"ContainerStarted","Data":"17b6b191b3196f755413f0edc86314530820822b4d582d2d02ab4c52da8baece"}
Nov 03 08:06:48.244284 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.244262    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2" event={"ID":"12307292-4369-4f20-9288-9b2005609416","Type":"ContainerStarted","Data":"8ebc9ee133798b1c6f28bb65926a3dad9df2da4754aebb4c84588ededc84e2c3"}
Nov 03 08:06:48.265725 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.265602    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5" podStartSLOduration=3.265580842 podStartE2EDuration="3.265580842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:48.196483703 +0000 UTC m=+806.310947635" watchObservedRunningTime="2026-11-03 08:06:48.265580842 +0000 UTC m=+806.380044773"
Nov 03 08:06:48.276782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.269421    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2" podStartSLOduration=3.269408273 podStartE2EDuration="3.269408273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:06:48.213806278 +0000 UTC m=+806.328270210" watchObservedRunningTime="2026-11-03 08:06:48.269408273 +0000 UTC m=+806.383872207"
Nov 03 08:06:48.316795 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.315835    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:06:48.316795 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.315981    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 08:06:48.316795 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.316029    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2" podUID="12307292-4369-4f20-9288-9b2005609416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 08:06:48.373539 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.373106    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:06:48.373539 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.373232    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 08:06:48.373539 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.373273    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5" podUID="d1a356ec-fba8-45e1-87ba-13950f3e1fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 08:06:48.539843 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.537717    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:48.679459 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.679414    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:06:48.944382 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.944328    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:06:48.987229 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:48.987187    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:06:49.092929 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:49.089888    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 08:06:49.092929 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:49.089949    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 08:06:49.104169 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:49.104130    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:06:49.104858 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:49.104381    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:06:49.117779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:49.105701    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 08:06:49.118955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:49.118927    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 08:06:49.119191 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:49.119126    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:49.155583 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:49.155391    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded"
Nov 03 08:06:49.155583 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:49.155507    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 08:06:49.155583 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:49.155539    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr" podUID="64c976ca-8c6d-4835-b5ca-ba88be627ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 08:06:49.211845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:49.186961    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 08:06:49.212137 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:49.212098    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 08:06:49.328924 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:49.328886    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 08:06:49.329212 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:49.329178    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2" podUID="12307292-4369-4f20-9288-9b2005609416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 08:06:49.371385 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:49.371347    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 08:06:49.371663 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:49.371632    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5" podUID="d1a356ec-fba8-45e1-87ba-13950f3e1fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 08:06:49.550400 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:49.550295    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:49.551410 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:49.551385    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:06:50.048011 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.047961    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:50.099655 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.099604    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:50.131783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.128112    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:50.191783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.188066    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:50.218228 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.218179    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:06:50.670355 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.670302    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:06:50.767785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.766580    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j" podUID="67eb2350-8895-4e7d-b1f2-e78e1ff5cf23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:50.833812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.791893    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn" podUID="6960b432-0c5c-4c03-bcc5-dec9ff9466c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:50.833812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.792200    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz" podUID="33f324bd-79b1-4540-a090-7e4a9d92f10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 03 08:06:50.833812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.801014    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:06:50.833812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.825593    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 08:06:50.833812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.825651    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 08:06:50.833812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.825726    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 08:06:50.833812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.825748    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 08:06:50.835726 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.835680    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": context deadline exceeded"
Nov 03 08:06:50.856337 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.850298    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 08:06:50.856337 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.850349    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2" podUID="12307292-4369-4f20-9288-9b2005609416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 08:06:50.968013 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.905893    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 08:06:50.968013 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.905955    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5" podUID="d1a356ec-fba8-45e1-87ba-13950f3e1fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 08:06:50.968013 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.920615    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 08:06:50.968013 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.920665    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr" podUID="64c976ca-8c6d-4835-b5ca-ba88be627ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 08:06:50.968013 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.924389    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded"
Nov 03 08:06:50.991150 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.990933    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:51.015788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:50.993596    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:51.128090 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.128028    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:51.152999 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.152943    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:06:51.184817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.174476    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 08:06:51.184817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.174529    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr" podUID="64c976ca-8c6d-4835-b5ca-ba88be627ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 08:06:51.186927 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.186896    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 08:06:51.187133 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.187107    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 08:06:51.187274 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.062848    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:06:51.201016 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.200991    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 08:06:51.201225 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.201200    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 08:06:51.230123 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.230036    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:06:51.301325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.301289    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 08:06:51.301612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.301570    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 08:06:51.302466 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.302429    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 08:06:51.302645 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.302625    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 08:06:51.365007 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.364974    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 08:06:51.368899 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.368858    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2" podUID="12307292-4369-4f20-9288-9b2005609416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 08:06:51.369377 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.369350    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 08:06:51.369536 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.369516    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5" podUID="d1a356ec-fba8-45e1-87ba-13950f3e1fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 08:06:51.369685 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.369667    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:51.383276 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:06:51.383250    5070 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.361s"
Nov 03 08:06:51.391988 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.391965    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:06:51.433784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.408943    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:06:51.435196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.435170    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:06:51.435417 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.435401    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:06:51.435616 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.435602    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:06:51.435748 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.435736    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:06:51.480952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.480868    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:06:51.506354 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.393984    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6"
Nov 03 08:06:51.755857 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.742136    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:51.875164 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:51.834831    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:06:52.094840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.033800    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:06:52.094840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.034224    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:06:52.094840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.091161    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 08:06:52.094840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.091214    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 08:06:52.119784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.117367    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 08:06:52.119784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.117422    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 08:06:52.119784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.117484    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:52.161999 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.161957    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:52.205828 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.190039    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded"
Nov 03 08:06:52.205828 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.191204    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:52.205828 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.192344    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:52.205828 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.193011    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 08:06:52.205828 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.193048    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr" podUID="64c976ca-8c6d-4835-b5ca-ba88be627ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 08:06:52.205828 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.193107    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 08:06:52.205828 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.193131    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 08:06:52.229812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.228442    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48af7f66-65e9-4ef0-b99a-7046e3303bef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48af7f66-65e9-4ef0-b99a-7046e3303bef-maas-api-xks-ca\") pod \"48af7f66-65e9-4ef0-b99a-7046e3303bef\" (UID: \"48af7f66-65e9-4ef0-b99a-7046e3303bef\") "
Nov 03 08:06:52.229812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.228714    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48af7f66-65e9-4ef0-b99a-7046e3303bef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48af7f66-65e9-4ef0-b99a-7046e3303bef-maas-api-openshift-ca\") pod \"48af7f66-65e9-4ef0-b99a-7046e3303bef\" (UID: \"48af7f66-65e9-4ef0-b99a-7046e3303bef\") "
Nov 03 08:06:52.229812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.228891    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48af7f66-65e9-4ef0-b99a-7046e3303bef-kube-api-access-b5g95\" (UniqueName: \"kubernetes.io/projected/48af7f66-65e9-4ef0-b99a-7046e3303bef-kube-api-access-b5g95\") pod \"48af7f66-65e9-4ef0-b99a-7046e3303bef\" (UID: \"48af7f66-65e9-4ef0-b99a-7046e3303bef\") "
Nov 03 08:06:52.327794 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.326850    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:52.519782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.519547    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48af7f66-65e9-4ef0-b99a-7046e3303bef-maas-api-openshift-ca" pod "48af7f66-65e9-4ef0-b99a-7046e3303bef" (UID: "48af7f66-65e9-4ef0-b99a-7046e3303bef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:06:52.606607 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.577086    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48af7f66-65e9-4ef0-b99a-7046e3303bef-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:52.978887 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.625812    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:52.991789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.768373    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 08:06:53.037785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.993073    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2" podUID="12307292-4369-4f20-9288-9b2005609416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 08:06:53.102874 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.917269    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:53.103398 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.913772    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:53.103819 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.768428    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 08:06:53.103999 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:53.103971    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5" podUID="d1a356ec-fba8-45e1-87ba-13950f3e1fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 08:06:53.170364 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:53.039977    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:06:53.181787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.992089    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48af7f66-65e9-4ef0-b99a-7046e3303bef-maas-api-xks-ca" pod "48af7f66-65e9-4ef0-b99a-7046e3303bef" (UID: "48af7f66-65e9-4ef0-b99a-7046e3303bef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:06:53.182105 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:52.992183    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48af7f66-65e9-4ef0-b99a-7046e3303bef-kube-api-access-b5g95" pod "48af7f66-65e9-4ef0-b99a-7046e3303bef" (UID: "48af7f66-65e9-4ef0-b99a-7046e3303bef"). InnerVolumeSpecName "kube-api-access-b5g95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:06:53.182202 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:53.098584    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 08:06:53.185784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:53.182314    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 08:06:53.186000 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:53.120067    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 08:06:53.186124 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:53.186103    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 08:06:53.190784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:53.120113    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:06:53.191243 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:06:53.121379    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:06:53.275855 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:53.259010    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 08:06:53.275855 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:53.259073    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr" podUID="64c976ca-8c6d-4835-b5ca-ba88be627ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 08:06:53.462798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:53.461304    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:06:53.725790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:53.715152    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:06:53.768785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:53.748064    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:06:53.860095 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:53.860060    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 08:06:53.860408 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:53.860362    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 08:06:53.889786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:53.874505    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 08:06:53.957633 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:53.957589    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5" podUID="d1a356ec-fba8-45e1-87ba-13950f3e1fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 08:06:53.965911 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:53.934465    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5g95\" (UniqueName: \"kubernetes.io/projected/48af7f66-65e9-4ef0-b99a-7046e3303bef-kube-api-access-b5g95\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:54.113789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.102571    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48af7f66-65e9-4ef0-b99a-7046e3303bef-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:06:54.544847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.024051    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:54.610789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.068101    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 08:06:54.610789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.595616    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2" podUID="12307292-4369-4f20-9288-9b2005609416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 08:06:54.610789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.102414    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded"
Nov 03 08:06:54.610789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.265854    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 08:06:54.610789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.596121    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 08:06:54.610789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.209069    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:06:54.610789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.102541    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:06:54.610789 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:06:54.154091    5070 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.118s"
Nov 03 08:06:54.623781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.612797    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 08:06:54.624092 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.624062    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 08:06:54.624277 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.534262    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:54.723158 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:53.981016    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6"
Nov 03 08:06:54.723425 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.500984    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 08:06:54.723609 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.723573    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 08:06:54.753186 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.521286    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:06:54.753453 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.488863    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:06:55.120787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.521195    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:06:55.120787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.722995    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:06:55.120787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.237671    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:06:55.120787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.534018    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 08:06:55.120787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.754381    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:55.120787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.676083    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 08:06:55.120787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:55.111725    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr" podUID="64c976ca-8c6d-4835-b5ca-ba88be627ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 08:06:55.120787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:55.111792    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5" podUID="d1a356ec-fba8-45e1-87ba-13950f3e1fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 08:06:55.120787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:54.851532    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 08:06:55.120787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:55.111913    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2" podUID="12307292-4369-4f20-9288-9b2005609416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 08:06:55.328925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:55.328882    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 08:06:55.329228 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:55.329193    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 08:06:55.350962 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:55.350911    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:55.375837 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:55.373136    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 08:06:55.375837 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:55.373187    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5" podUID="d1a356ec-fba8-45e1-87ba-13950f3e1fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 08:06:55.424786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:55.417002    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 08:06:55.458795 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:55.451153    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 08:06:55.485042 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:55.480865    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 08:06:55.485251 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:55.485226    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr" podUID="64c976ca-8c6d-4835-b5ca-ba88be627ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 08:06:55.508715 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:55.508682    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 08:06:55.509032 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:55.508988    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 08:06:55.513979 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:55.439886    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 08:06:55.519727 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:55.519693    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2" podUID="12307292-4369-4f20-9288-9b2005609416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 08:06:55.607796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:55.599404    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:55.607796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:55.599770    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:06:55.607796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:55.600031    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:06:55.699838 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:06:55.672556    5070 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.076s"
Nov 03 08:06:55.724784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:55.700403    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6" event={"ID":"48af7f66-65e9-4ef0-b99a-7046e3303bef","Type":"ContainerDied","Data":"0155fcb56f781150687d000a84f9351a4578e7abe4180cc8138466aada451f6a"}
Nov 03 08:06:55.724784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:55.708929    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0155fcb56f781150687d000a84f9351a4578e7abe4180cc8138466aada451f6a"
Nov 03 08:06:56.071832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.071713    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:06:56.143959 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.128430    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 08:06:56.143959 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.128488    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 08:06:56.169710 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.169675    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 08:06:56.169931 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.169734    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr" podUID="64c976ca-8c6d-4835-b5ca-ba88be627ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 08:06:56.193831 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.193798    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 08:06:56.194127 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.194079    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 08:06:56.437083 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.437030    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:56.458003 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.457963    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:06:56.538783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.458259    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:56.538783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.458287    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:56.538783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.487271    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:06:56.538783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.516979    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:56.538783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.517336    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 08:06:56.538783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.538143    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5" podUID="d1a356ec-fba8-45e1-87ba-13950f3e1fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 08:06:56.538783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.517464    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 08:06:56.538783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.538207    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2" podUID="12307292-4369-4f20-9288-9b2005609416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 08:06:56.598786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.565645    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 08:06:56.612809 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.609895    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 08:06:56.613297 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.612840    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:56.638813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.615114    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:06:56.679917 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.644812    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:06:56.871561 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:56.871505    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:57.083797 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:57.077320    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:57.112796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:57.109790    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 08:06:57.112796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:57.109849    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 08:06:57.112796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:57.110122    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 08:06:57.112796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:57.110154    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 08:06:57.165732 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:57.165637    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 08:06:57.166026 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:57.165979    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr" podUID="64c976ca-8c6d-4835-b5ca-ba88be627ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 08:06:57.169232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:57.169199    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 08:06:57.172780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:57.169384    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 08:06:57.270786 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:06:57.246304    5070 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.174s"
Nov 03 08:06:57.368886 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:57.368850    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 08:06:57.401061 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:57.401018    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2" podUID="12307292-4369-4f20-9288-9b2005609416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 08:06:57.401320 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:57.370798    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 08:06:57.401463 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:57.401439    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5" podUID="d1a356ec-fba8-45e1-87ba-13950f3e1fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 08:06:57.439024 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:57.438916    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:57.556824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:57.552375    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:06:57.556824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:57.552806    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:57.556824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:57.553234    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:06:57.613710 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:57.613653    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:57.615257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:57.615215    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:57.649082 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:57.649041    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:57.685779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:57.684692    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:06:58.129063 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:58.119216    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:58.248248 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:58.188794    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 08:06:58.248248 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:58.188849    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 08:06:58.248248 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:58.189379    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 08:06:58.248248 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:58.189418    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 08:06:58.248248 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:58.196492    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Nov 03 08:06:58.248248 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:58.196551    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Nov 03 08:06:58.248248 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:58.234730    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 08:06:58.248248 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:58.236021    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr" podUID="64c976ca-8c6d-4835-b5ca-ba88be627ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 08:06:58.315521 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:58.315453    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 08:06:58.316216 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:58.316182    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2" podUID="12307292-4369-4f20-9288-9b2005609416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 08:06:58.368446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:58.368411    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 08:06:58.368666 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:58.368644    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5" podUID="d1a356ec-fba8-45e1-87ba-13950f3e1fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 08:06:58.470659 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:58.470559    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:58.555989 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:58.555928    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:58.556646 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:58.556608    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:06:58.557146 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:58.557111    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:06:58.615663 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:58.615514    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:58.616291 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:58.615601    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:58.669250 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:58.660024    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:58.712030 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:58.711991    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:06:59.062147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:59.014273    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:06:59.146789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:59.146170    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:06:59.157110 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:59.151428    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:06:59.157110 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:59.153844    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:59.159521 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:59.159425    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body=
Nov 03 08:06:59.159521 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:59.159466    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr" podUID="64c976ca-8c6d-4835-b5ca-ba88be627ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused"
Nov 03 08:06:59.213780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:59.188601    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:06:59.339838 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:59.314166    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Nov 03 08:06:59.353977 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:59.353862    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2" podUID="12307292-4369-4f20-9288-9b2005609416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Nov 03 08:06:59.473133 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:59.429922    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 08:06:59.473133 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:59.429982    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5" podUID="d1a356ec-fba8-45e1-87ba-13950f3e1fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 08:06:59.476044 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:59.473607    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:06:59.637194 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:59.631932    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:06:59.677811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:59.677776    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:06:59.703877 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:59.703836    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:06:59.706321 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:59.705402    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": context deadline exceeded"
Nov 03 08:06:59.707511 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:59.707477    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:06:59.740041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:59.739989    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:06:59.908358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:06:59.908248    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:07:00.222059 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:00.221963    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:00.222314 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:00.222281    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:00.223482 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:00.223449    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:00.223659 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:00.223638    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:00.252718 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:00.251877    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:00.252718 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:00.251934    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:00.514226 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:00.514119    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:07:00.633691 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:00.633645    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:07:00.678871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:00.678829    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:00.715090 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:00.713684    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:00.715090 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:00.714473    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:00.764887 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:00.744261    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:07:00.764887 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:00.745875    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:07:00.764887 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:00.764227    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:07:00.802279 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:00.780907    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:00.921814 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:00.885452    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded"
Nov 03 08:07:01.129783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.126419    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:01.129783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.127635    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:01.149813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.142864    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"]
Nov 03 08:07:01.149813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.143946    5070 cpu_manager.go:414] "containerMap: removing container" podUID="48af7f66-65e9-4ef0-b99a-7046e3303bef" containerName="revoke-keys"
Nov 03 08:07:01.149813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.143965    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48af7f66-65e9-4ef0-b99a-7046e3303bef" containerName="revoke-keys"
Nov 03 08:07:01.149813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.144161    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="48af7f66-65e9-4ef0-b99a-7046e3303bef" containerName="revoke-keys"
Nov 03 08:07:01.198782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.173191    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:07:01.199153 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.199129    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:07:01.199321 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.199306    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:07:01.265783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.175892    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.279847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.215844    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:01.310782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.282836    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:01.310782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.282895    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:01.310782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.286820    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:01.310782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.286863    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:01.310782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.288670    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"]
Nov 03 08:07:01.310782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.296114    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:01.310782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.296158    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:01.311940 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.311439    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:07:01.362916 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.354853    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:01.362916 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.354915    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2" podUID="12307292-4369-4f20-9288-9b2005609416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:01.449832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.367689    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.450138 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.450107    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.450342 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.450323    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f76ed594-a720-4470-85da-02885a26dd4c-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.450574 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.450553    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.450727 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.450708    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj8sq\" (UniqueName: \"kubernetes.io/projected/f76ed594-a720-4470-85da-02885a26dd4c-kube-api-access-vj8sq\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.450911 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.450892    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f76ed594-a720-4470-85da-02885a26dd4c-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.451286 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.451265    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f76ed594-a720-4470-85da-02885a26dd4c-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.451487 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.451466    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.451740 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.451716    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.451980 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.383145    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:07:01.471787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.412905    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:01.471787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.464954    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5" podUID="d1a356ec-fba8-45e1-87ba-13950f3e1fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:01.516741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.516698    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:01.561787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.553845    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.575907 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.575865    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.576260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.576238    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f76ed594-a720-4470-85da-02885a26dd4c-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.576436 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.576417    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.576581 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.576562    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj8sq\" (UniqueName: \"kubernetes.io/projected/f76ed594-a720-4470-85da-02885a26dd4c-kube-api-access-vj8sq\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.576737 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.576721    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f76ed594-a720-4470-85da-02885a26dd4c-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.582138 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.582109    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f76ed594-a720-4470-85da-02885a26dd4c-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.599922 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.599877    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.600188 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.600163    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.600906 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.600880    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.639820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.601317    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.639820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.602488    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.639820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.611455    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f76ed594-a720-4470-85da-02885a26dd4c-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.668531 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.659958    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:07:01.668531 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.660650    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:07:01.680794 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.679774    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f76ed594-a720-4470-85da-02885a26dd4c-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.680794 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.679867    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:07:01.761196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.720145    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:07:01.761196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.734229    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:01.762167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.761855    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.841588 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.794864    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:07:01.859035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.843428    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj8sq\" (UniqueName: \"kubernetes.io/projected/f76ed594-a720-4470-85da-02885a26dd4c-kube-api-access-vj8sq\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.890502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.890464    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:01.964210 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:01.962721    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f76ed594-a720-4470-85da-02885a26dd4c-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:02.138824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.129384    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:07:02.138824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.132447    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:02.183781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.169370    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:02.196417 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.196379    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:02.254782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.235201    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:07:02.304788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.281639    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:07:02.328791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.316844    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:02.328791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.316913    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:02.343618 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.343480    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:02.344166 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.344044    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:02.495841 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.470859    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:02.495841 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.470943    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:02.548356 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.536455    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:02.567799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.566414    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:02.567799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.566497    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2" podUID="12307292-4369-4f20-9288-9b2005609416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:02.618287 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.598576    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:02.618287 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.598648    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5" podUID="d1a356ec-fba8-45e1-87ba-13950f3e1fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:02.745031 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.744957    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:07:02.835963 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.835853    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:02.836819 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.836785    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:07:02.899199 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.811855    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:02.899724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.859603    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:07:02.977784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.947706    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:07:02.978430 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:02.978402    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:07:03.275789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:03.253463    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:03.304018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:03.303966    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:03.314487 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:03.314449    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:03.364947 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:03.364909    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:03.369790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:03.365893    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:03.413070 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:03.411199    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:07:03.414735 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:03.414715    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:07:03.428866 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:03.416293    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:03.428866 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:03.416345    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:03.462808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:03.448860    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:07:03.526224 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:03.526138    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:07:03.696768 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:03.544284    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:03.697718 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:03.567903    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:03.697996 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:03.697949    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:03.778025 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:03.777933    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:04.239689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:04.063752    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:04.240829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:04.069321    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:04.241008 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:04.117868    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:07:04.242151 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:04.117913    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:07:04.263897 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:04.263857    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:04.265254 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:04.119616    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:04.266467 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:04.266434    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:04.311964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:04.311910    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:04.342263 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:04.342223    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:04.528980 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:04.528890    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:04.529241 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:04.529208    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:04.564508 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:04.564468    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:04.564792 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:04.564743    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:04.728785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:04.720320    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:07:04.761947 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:04.761872    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:04.762236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:04.762175    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:05.274494 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.274433    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:05.522137 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.275100    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:07:05.522938 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.275148    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:07:05.523325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.275168    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:07:05.523692 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.275186    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:07:05.524051 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.322979    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:05.527815 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.347456    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:07:05.549523 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.347552    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:05.550025 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.348214    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:05.630057 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.620634    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:05.630057 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.620692    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:05.657043 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.657005    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:05.664895 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.664841    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:05.731936 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.731705    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:07:05.787152 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:07:05.749673    5070 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.627s"
Nov 03 08:07:05.787152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.779819    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx"]
Nov 03 08:07:05.792562 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.792539    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx"
Nov 03 08:07:05.797674 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.797649    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:07:05.797974 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.797958    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:07:05.798175 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.798082    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:07:05.798354 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.798340    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:07:05.813295 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.813257    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:05.813581 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.813558    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:05.823586 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.823219    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx"]
Nov 03 08:07:05.892651 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.891856    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:07:05.906974 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:05.906949    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:07:06.279775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.274415    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab19328-5bd5-40cc-b030-af55b89d76ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b1fce47-phkkx\" (UID: \"3ab19328-5bd5-40cc-b030-af55b89d76ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx"
Nov 03 08:07:06.279775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.275816    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpb67\" (UniqueName: \"kubernetes.io/projected/3ab19328-5bd5-40cc-b030-af55b89d76ba-kube-api-access-xpb67\") pod \"maas-api-revoke-sub-3b1fce47-phkkx\" (UID: \"3ab19328-5bd5-40cc-b030-af55b89d76ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx"
Nov 03 08:07:06.279775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.275930    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab19328-5bd5-40cc-b030-af55b89d76ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b1fce47-phkkx\" (UID: \"3ab19328-5bd5-40cc-b030-af55b89d76ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx"
Nov 03 08:07:06.301604 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.301573    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:07:06.302062 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.302042    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:07:06.302283 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.302268    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:07:06.302425 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.302410    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"]
Nov 03 08:07:06.324549 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.315943    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:07:06.324549 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.322691    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:07:06.400384 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.400355    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:07:06.406861 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.406741    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab19328-5bd5-40cc-b030-af55b89d76ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b1fce47-phkkx\" (UID: \"3ab19328-5bd5-40cc-b030-af55b89d76ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx"
Nov 03 08:07:06.407009 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.406913    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpb67\" (UniqueName: \"kubernetes.io/projected/3ab19328-5bd5-40cc-b030-af55b89d76ba-kube-api-access-xpb67\") pod \"maas-api-revoke-sub-3b1fce47-phkkx\" (UID: \"3ab19328-5bd5-40cc-b030-af55b89d76ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx"
Nov 03 08:07:06.407083 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.407054    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab19328-5bd5-40cc-b030-af55b89d76ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b1fce47-phkkx\" (UID: \"3ab19328-5bd5-40cc-b030-af55b89d76ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx"
Nov 03 08:07:06.414742 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.409139    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:07:06.414742 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.410288    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:07:06.414742 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.413173    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab19328-5bd5-40cc-b030-af55b89d76ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b1fce47-phkkx\" (UID: \"3ab19328-5bd5-40cc-b030-af55b89d76ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx"
Nov 03 08:07:06.438988 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.438740    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab19328-5bd5-40cc-b030-af55b89d76ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b1fce47-phkkx\" (UID: \"3ab19328-5bd5-40cc-b030-af55b89d76ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx"
Nov 03 08:07:06.441458 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.441437    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:07:06.447305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.447285    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:07:06.455822 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.451695    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:07:06.455822 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.451788    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:07:06.455822 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.451843    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:07:06.490687 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.490589    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:07:06.512630 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.499489    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:07:06.542113 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.541644    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:07:06.571869 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.571801    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpb67\" (UniqueName: \"kubernetes.io/projected/3ab19328-5bd5-40cc-b030-af55b89d76ba-kube-api-access-xpb67\") pod \"maas-api-revoke-sub-3b1fce47-phkkx\" (UID: \"3ab19328-5bd5-40cc-b030-af55b89d76ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx"
Nov 03 08:07:06.803470 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.803264    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx"
Nov 03 08:07:06.805921 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.805827    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s" event={"ID":"f76ed594-a720-4470-85da-02885a26dd4c","Type":"ContainerStarted","Data":"9246caf5c83ab92efbfc6b529ad8a1e82cc5ceac0aa3bb0553d22249facf45c3"}
Nov 03 08:07:06.805921 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.805879    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s" event={"ID":"f76ed594-a720-4470-85da-02885a26dd4c","Type":"ContainerStarted","Data":"6289c0a2291092624b1bcb66ff3d67c84dc4d7b7b2702abdcda746969321a91d"}
Nov 03 08:07:06.827264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:06.827217    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s" podStartSLOduration=6.827202144 podStartE2EDuration="6.827202144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:06.822057745 +0000 UTC m=+824.936521677" watchObservedRunningTime="2026-11-03 08:07:06.827202144 +0000 UTC m=+824.941666165"
Nov 03 08:07:07.098723 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:07:07.098693    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab19328_5bd5_40cc_b030_af55b89d76ba.slice/crio-ca9fbf17bae4c04d96b5903ed4e218a5ce1f262cc13fe4517b6a71fb1c723c7a WatchSource:0}: Error finding container ca9fbf17bae4c04d96b5903ed4e218a5ce1f262cc13fe4517b6a71fb1c723c7a: Status 404 returned error can't find the container with id ca9fbf17bae4c04d96b5903ed4e218a5ce1f262cc13fe4517b6a71fb1c723c7a
Nov 03 08:07:07.103605 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:07.103578    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx"]
Nov 03 08:07:07.219494 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:07.219466    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:07.220659 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:07.220629    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 08:07:07.220789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:07.220681    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s" podUID="f76ed594-a720-4470-85da-02885a26dd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 08:07:07.814986 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:07.814891    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx" event={"ID":"3ab19328-5bd5-40cc-b030-af55b89d76ba","Type":"ContainerStarted","Data":"68c58bc4981f62f88dda731953e87637bbe5a96065fdcc9320558bab97193543"}
Nov 03 08:07:07.814986 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:07.814948    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx" event={"ID":"3ab19328-5bd5-40cc-b030-af55b89d76ba","Type":"ContainerStarted","Data":"ca9fbf17bae4c04d96b5903ed4e218a5ce1f262cc13fe4517b6a71fb1c723c7a"}
Nov 03 08:07:07.840161 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:07.835199    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx" podStartSLOduration=2.835182321 podStartE2EDuration="2.835182321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:07.831231594 +0000 UTC m=+825.945695522" watchObservedRunningTime="2026-11-03 08:07:07.835182321 +0000 UTC m=+825.949646252"
Nov 03 08:07:08.219607 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:08.219569    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 08:07:08.219793 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:08.219625    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s" podUID="f76ed594-a720-4470-85da-02885a26dd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 08:07:08.238594 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:08.238562    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw"]
Nov 03 08:07:08.243860 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:08.243446    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw"
Nov 03 08:07:08.253083 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:08.252644    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw"]
Nov 03 08:07:08.353776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:08.353714    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb1f4c17-b03f-4c23-9742-1869aeb79837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-408f9b14-m2mbw\" (UID: \"bb1f4c17-b03f-4c23-9742-1869aeb79837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw"
Nov 03 08:07:08.353959 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:08.353900    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftncp\" (UniqueName: \"kubernetes.io/projected/bb1f4c17-b03f-4c23-9742-1869aeb79837-kube-api-access-ftncp\") pod \"maas-api-revoke-sub-408f9b14-m2mbw\" (UID: \"bb1f4c17-b03f-4c23-9742-1869aeb79837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw"
Nov 03 08:07:08.354126 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:08.354099    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb1f4c17-b03f-4c23-9742-1869aeb79837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-408f9b14-m2mbw\" (UID: \"bb1f4c17-b03f-4c23-9742-1869aeb79837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw"
Nov 03 08:07:08.464781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:08.456912    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftncp\" (UniqueName: \"kubernetes.io/projected/bb1f4c17-b03f-4c23-9742-1869aeb79837-kube-api-access-ftncp\") pod \"maas-api-revoke-sub-408f9b14-m2mbw\" (UID: \"bb1f4c17-b03f-4c23-9742-1869aeb79837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw"
Nov 03 08:07:08.464781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:08.457162    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb1f4c17-b03f-4c23-9742-1869aeb79837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-408f9b14-m2mbw\" (UID: \"bb1f4c17-b03f-4c23-9742-1869aeb79837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw"
Nov 03 08:07:08.464781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:08.457403    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb1f4c17-b03f-4c23-9742-1869aeb79837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-408f9b14-m2mbw\" (UID: \"bb1f4c17-b03f-4c23-9742-1869aeb79837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw"
Nov 03 08:07:08.464781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:08.458383    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb1f4c17-b03f-4c23-9742-1869aeb79837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-408f9b14-m2mbw\" (UID: \"bb1f4c17-b03f-4c23-9742-1869aeb79837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw"
Nov 03 08:07:08.511635 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:08.511542    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftncp\" (UniqueName: \"kubernetes.io/projected/bb1f4c17-b03f-4c23-9742-1869aeb79837-kube-api-access-ftncp\") pod \"maas-api-revoke-sub-408f9b14-m2mbw\" (UID: \"bb1f4c17-b03f-4c23-9742-1869aeb79837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw"
Nov 03 08:07:08.512454 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:08.512398    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb1f4c17-b03f-4c23-9742-1869aeb79837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-408f9b14-m2mbw\" (UID: \"bb1f4c17-b03f-4c23-9742-1869aeb79837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw"
Nov 03 08:07:08.622414 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:08.622383    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw"
Nov 03 08:07:08.832827 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:08.832721    5070 generic.go:356] "Generic (PLEG): container finished" podID="3ab19328-5bd5-40cc-b030-af55b89d76ba" containerID="68c58bc4981f62f88dda731953e87637bbe5a96065fdcc9320558bab97193543" exitCode=0
Nov 03 08:07:08.832827 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:08.832805    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx" event={"ID":"3ab19328-5bd5-40cc-b030-af55b89d76ba","Type":"ContainerDied","Data":"68c58bc4981f62f88dda731953e87637bbe5a96065fdcc9320558bab97193543"}
Nov 03 08:07:08.938219 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:07:08.938182    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1f4c17_b03f_4c23_9742_1869aeb79837.slice/crio-112c172899b8716906d035bedfcc074d5578923d6b832cb0eeb0f0ecd0df1116 WatchSource:0}: Error finding container 112c172899b8716906d035bedfcc074d5578923d6b832cb0eeb0f0ecd0df1116: Status 404 returned error can't find the container with id 112c172899b8716906d035bedfcc074d5578923d6b832cb0eeb0f0ecd0df1116
Nov 03 08:07:08.945249 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:08.945221    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw"]
Nov 03 08:07:09.222780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:09.221733    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 08:07:09.222780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:09.221820    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s" podUID="f76ed594-a720-4470-85da-02885a26dd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 08:07:09.843645 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:09.843562    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw" event={"ID":"bb1f4c17-b03f-4c23-9742-1869aeb79837","Type":"ContainerStarted","Data":"44d3dd8c9b947a32810f61647905a1f5cbf21ef1fbb2f1236cb251e7daaee94c"}
Nov 03 08:07:09.843645 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:09.843615    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw" event={"ID":"bb1f4c17-b03f-4c23-9742-1869aeb79837","Type":"ContainerStarted","Data":"112c172899b8716906d035bedfcc074d5578923d6b832cb0eeb0f0ecd0df1116"}
Nov 03 08:07:09.891780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:09.890008    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw" podStartSLOduration=1.8899948389999999 podStartE2EDuration="1.889994839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:09.880869258 +0000 UTC m=+827.995333189" watchObservedRunningTime="2026-11-03 08:07:09.889994839 +0000 UTC m=+828.004458793"
Nov 03 08:07:09.940439 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:09.937848    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl"]
Nov 03 08:07:09.985823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:09.947391    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl"
Nov 03 08:07:09.985823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:09.954079    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl"]
Nov 03 08:07:10.104915 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.104878    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd8c4\" (UniqueName: \"kubernetes.io/projected/a919ae58-9b50-4450-a523-1f64f89433c0-kube-api-access-fd8c4\") pod \"maas-api-revoke-sub-5a29cf07-7phcl\" (UID: \"a919ae58-9b50-4450-a523-1f64f89433c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl"
Nov 03 08:07:10.105057 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.104963    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a919ae58-9b50-4450-a523-1f64f89433c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29cf07-7phcl\" (UID: \"a919ae58-9b50-4450-a523-1f64f89433c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl"
Nov 03 08:07:10.105124 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.105093    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a919ae58-9b50-4450-a523-1f64f89433c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29cf07-7phcl\" (UID: \"a919ae58-9b50-4450-a523-1f64f89433c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl"
Nov 03 08:07:10.190423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.189970    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx"
Nov 03 08:07:10.231791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.211940    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd8c4\" (UniqueName: \"kubernetes.io/projected/a919ae58-9b50-4450-a523-1f64f89433c0-kube-api-access-fd8c4\") pod \"maas-api-revoke-sub-5a29cf07-7phcl\" (UID: \"a919ae58-9b50-4450-a523-1f64f89433c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl"
Nov 03 08:07:10.231791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.212007    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a919ae58-9b50-4450-a523-1f64f89433c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29cf07-7phcl\" (UID: \"a919ae58-9b50-4450-a523-1f64f89433c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl"
Nov 03 08:07:10.231791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.212092    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a919ae58-9b50-4450-a523-1f64f89433c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29cf07-7phcl\" (UID: \"a919ae58-9b50-4450-a523-1f64f89433c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl"
Nov 03 08:07:10.231791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.217862    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a919ae58-9b50-4450-a523-1f64f89433c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29cf07-7phcl\" (UID: \"a919ae58-9b50-4450-a523-1f64f89433c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl"
Nov 03 08:07:10.245158 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.245089    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a919ae58-9b50-4450-a523-1f64f89433c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29cf07-7phcl\" (UID: \"a919ae58-9b50-4450-a523-1f64f89433c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl"
Nov 03 08:07:10.245352 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.245313    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 08:07:10.245431 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.245403    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd8c4\" (UniqueName: \"kubernetes.io/projected/a919ae58-9b50-4450-a523-1f64f89433c0-kube-api-access-fd8c4\") pod \"maas-api-revoke-sub-5a29cf07-7phcl\" (UID: \"a919ae58-9b50-4450-a523-1f64f89433c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl"
Nov 03 08:07:10.245431 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.245408    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s" podUID="f76ed594-a720-4470-85da-02885a26dd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 08:07:10.296305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.296235    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl"
Nov 03 08:07:10.312892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.312853    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ab19328-5bd5-40cc-b030-af55b89d76ba-kube-api-access-xpb67\" (UniqueName: \"kubernetes.io/projected/3ab19328-5bd5-40cc-b030-af55b89d76ba-kube-api-access-xpb67\") pod \"3ab19328-5bd5-40cc-b030-af55b89d76ba\" (UID: \"3ab19328-5bd5-40cc-b030-af55b89d76ba\") "
Nov 03 08:07:10.313020 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.312949    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ab19328-5bd5-40cc-b030-af55b89d76ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab19328-5bd5-40cc-b030-af55b89d76ba-maas-api-xks-ca\") pod \"3ab19328-5bd5-40cc-b030-af55b89d76ba\" (UID: \"3ab19328-5bd5-40cc-b030-af55b89d76ba\") "
Nov 03 08:07:10.313268 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.313244    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ab19328-5bd5-40cc-b030-af55b89d76ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab19328-5bd5-40cc-b030-af55b89d76ba-maas-api-openshift-ca\") pod \"3ab19328-5bd5-40cc-b030-af55b89d76ba\" (UID: \"3ab19328-5bd5-40cc-b030-af55b89d76ba\") "
Nov 03 08:07:10.314447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.314419    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ab19328-5bd5-40cc-b030-af55b89d76ba-maas-api-openshift-ca" pod "3ab19328-5bd5-40cc-b030-af55b89d76ba" (UID: "3ab19328-5bd5-40cc-b030-af55b89d76ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:07:10.317027 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.317000    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ab19328-5bd5-40cc-b030-af55b89d76ba-kube-api-access-xpb67" pod "3ab19328-5bd5-40cc-b030-af55b89d76ba" (UID: "3ab19328-5bd5-40cc-b030-af55b89d76ba"). InnerVolumeSpecName "kube-api-access-xpb67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:07:10.322142 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.322105    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ab19328-5bd5-40cc-b030-af55b89d76ba-maas-api-xks-ca" pod "3ab19328-5bd5-40cc-b030-af55b89d76ba" (UID: "3ab19328-5bd5-40cc-b030-af55b89d76ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:07:10.416255 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.416129    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpb67\" (UniqueName: \"kubernetes.io/projected/3ab19328-5bd5-40cc-b030-af55b89d76ba-kube-api-access-xpb67\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:10.416255 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.416179    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab19328-5bd5-40cc-b030-af55b89d76ba-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:10.416255 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.416195    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab19328-5bd5-40cc-b030-af55b89d76ba-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:10.610172 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:07:10.609137    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda919ae58_9b50_4450_a523_1f64f89433c0.slice/crio-a1cd9daac944e922eae8c1f67228b8d12e92992ea9dc96a3d05e9a046537f09c WatchSource:0}: Error finding container a1cd9daac944e922eae8c1f67228b8d12e92992ea9dc96a3d05e9a046537f09c: Status 404 returned error can't find the container with id a1cd9daac944e922eae8c1f67228b8d12e92992ea9dc96a3d05e9a046537f09c
Nov 03 08:07:10.610785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.610731    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl"]
Nov 03 08:07:10.853930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.853899    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx"
Nov 03 08:07:10.854316 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.853929    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx" event={"ID":"3ab19328-5bd5-40cc-b030-af55b89d76ba","Type":"ContainerDied","Data":"ca9fbf17bae4c04d96b5903ed4e218a5ce1f262cc13fe4517b6a71fb1c723c7a"}
Nov 03 08:07:10.854316 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.853975    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca9fbf17bae4c04d96b5903ed4e218a5ce1f262cc13fe4517b6a71fb1c723c7a"
Nov 03 08:07:10.855711 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.855690    5070 generic.go:356] "Generic (PLEG): container finished" podID="bb1f4c17-b03f-4c23-9742-1869aeb79837" containerID="44d3dd8c9b947a32810f61647905a1f5cbf21ef1fbb2f1236cb251e7daaee94c" exitCode=0
Nov 03 08:07:10.855834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.855813    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw" event={"ID":"bb1f4c17-b03f-4c23-9742-1869aeb79837","Type":"ContainerDied","Data":"44d3dd8c9b947a32810f61647905a1f5cbf21ef1fbb2f1236cb251e7daaee94c"}
Nov 03 08:07:10.881437 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:10.881397    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl" event={"ID":"a919ae58-9b50-4450-a523-1f64f89433c0","Type":"ContainerStarted","Data":"a1cd9daac944e922eae8c1f67228b8d12e92992ea9dc96a3d05e9a046537f09c"}
Nov 03 08:07:11.220309 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:11.220284    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 08:07:11.220428 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:11.220328    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s" podUID="f76ed594-a720-4470-85da-02885a26dd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 08:07:11.941185 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:11.941149    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl" event={"ID":"a919ae58-9b50-4450-a523-1f64f89433c0","Type":"ContainerStarted","Data":"aa46d4b6d698a4e7b7cc4a5ad699e84723f812e4ae848bb11fbbd9f6d866d3a8"}
Nov 03 08:07:11.961861 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:11.961695    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl" podStartSLOduration=2.961676196 podStartE2EDuration="2.961676196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:11.953482907 +0000 UTC m=+830.067946853" watchObservedRunningTime="2026-11-03 08:07:11.961676196 +0000 UTC m=+830.076140120"
Nov 03 08:07:12.030704 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.030670    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2c0e3b3-6r9nz"]
Nov 03 08:07:12.031366 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.031346    5070 cpu_manager.go:414] "containerMap: removing container" podUID="3ab19328-5bd5-40cc-b030-af55b89d76ba" containerName="revoke-keys"
Nov 03 08:07:12.031366 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.031363    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ab19328-5bd5-40cc-b030-af55b89d76ba" containerName="revoke-keys"
Nov 03 08:07:12.031542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.031487    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ab19328-5bd5-40cc-b030-af55b89d76ba" containerName="revoke-keys"
Nov 03 08:07:12.036246 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.036223    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c0e3b3-6r9nz"
Nov 03 08:07:12.040481 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.040453    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2c0e3b3-6r9nz"]
Nov 03 08:07:12.147933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.147224    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b9ce1ef-4f78-4fdf-b846-e209df28cafe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2c0e3b3-6r9nz\" (UID: \"9b9ce1ef-4f78-4fdf-b846-e209df28cafe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c0e3b3-6r9nz"
Nov 03 08:07:12.147933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.147311    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b9ce1ef-4f78-4fdf-b846-e209df28cafe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2c0e3b3-6r9nz\" (UID: \"9b9ce1ef-4f78-4fdf-b846-e209df28cafe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c0e3b3-6r9nz"
Nov 03 08:07:12.147933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.147680    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lwjr\" (UniqueName: \"kubernetes.io/projected/9b9ce1ef-4f78-4fdf-b846-e209df28cafe-kube-api-access-9lwjr\") pod \"maas-api-revoke-sub-a2c0e3b3-6r9nz\" (UID: \"9b9ce1ef-4f78-4fdf-b846-e209df28cafe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c0e3b3-6r9nz"
Nov 03 08:07:12.219500 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.219470    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 08:07:12.219630 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.219522    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s" podUID="f76ed594-a720-4470-85da-02885a26dd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 08:07:12.235463 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.235438    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw"
Nov 03 08:07:12.249002 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.248969    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b9ce1ef-4f78-4fdf-b846-e209df28cafe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2c0e3b3-6r9nz\" (UID: \"9b9ce1ef-4f78-4fdf-b846-e209df28cafe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c0e3b3-6r9nz"
Nov 03 08:07:12.249124 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.249077    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b9ce1ef-4f78-4fdf-b846-e209df28cafe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2c0e3b3-6r9nz\" (UID: \"9b9ce1ef-4f78-4fdf-b846-e209df28cafe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c0e3b3-6r9nz"
Nov 03 08:07:12.249190 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.249177    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lwjr\" (UniqueName: \"kubernetes.io/projected/9b9ce1ef-4f78-4fdf-b846-e209df28cafe-kube-api-access-9lwjr\") pod \"maas-api-revoke-sub-a2c0e3b3-6r9nz\" (UID: \"9b9ce1ef-4f78-4fdf-b846-e209df28cafe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c0e3b3-6r9nz"
Nov 03 08:07:12.250128 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.250092    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b9ce1ef-4f78-4fdf-b846-e209df28cafe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2c0e3b3-6r9nz\" (UID: \"9b9ce1ef-4f78-4fdf-b846-e209df28cafe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c0e3b3-6r9nz"
Nov 03 08:07:12.252621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.252562    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b9ce1ef-4f78-4fdf-b846-e209df28cafe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2c0e3b3-6r9nz\" (UID: \"9b9ce1ef-4f78-4fdf-b846-e209df28cafe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c0e3b3-6r9nz"
Nov 03 08:07:12.293441 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.293409    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lwjr\" (UniqueName: \"kubernetes.io/projected/9b9ce1ef-4f78-4fdf-b846-e209df28cafe-kube-api-access-9lwjr\") pod \"maas-api-revoke-sub-a2c0e3b3-6r9nz\" (UID: \"9b9ce1ef-4f78-4fdf-b846-e209df28cafe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c0e3b3-6r9nz"
Nov 03 08:07:12.350219 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.350192    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb1f4c17-b03f-4c23-9742-1869aeb79837-kube-api-access-ftncp\" (UniqueName: \"kubernetes.io/projected/bb1f4c17-b03f-4c23-9742-1869aeb79837-kube-api-access-ftncp\") pod \"bb1f4c17-b03f-4c23-9742-1869aeb79837\" (UID: \"bb1f4c17-b03f-4c23-9742-1869aeb79837\") "
Nov 03 08:07:12.350407 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.350257    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb1f4c17-b03f-4c23-9742-1869aeb79837-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb1f4c17-b03f-4c23-9742-1869aeb79837-maas-api-openshift-ca\") pod \"bb1f4c17-b03f-4c23-9742-1869aeb79837\" (UID: \"bb1f4c17-b03f-4c23-9742-1869aeb79837\") "
Nov 03 08:07:12.350407 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.350405    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb1f4c17-b03f-4c23-9742-1869aeb79837-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb1f4c17-b03f-4c23-9742-1869aeb79837-maas-api-xks-ca\") pod \"bb1f4c17-b03f-4c23-9742-1869aeb79837\" (UID: \"bb1f4c17-b03f-4c23-9742-1869aeb79837\") "
Nov 03 08:07:12.350601 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.350576    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb1f4c17-b03f-4c23-9742-1869aeb79837-maas-api-openshift-ca" pod "bb1f4c17-b03f-4c23-9742-1869aeb79837" (UID: "bb1f4c17-b03f-4c23-9742-1869aeb79837"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:07:12.350986 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.350968    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb1f4c17-b03f-4c23-9742-1869aeb79837-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:12.352290 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.352259    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1f4c17-b03f-4c23-9742-1869aeb79837-kube-api-access-ftncp" pod "bb1f4c17-b03f-4c23-9742-1869aeb79837" (UID: "bb1f4c17-b03f-4c23-9742-1869aeb79837"). InnerVolumeSpecName "kube-api-access-ftncp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:07:12.352290 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.352282    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb1f4c17-b03f-4c23-9742-1869aeb79837-maas-api-xks-ca" pod "bb1f4c17-b03f-4c23-9742-1869aeb79837" (UID: "bb1f4c17-b03f-4c23-9742-1869aeb79837"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:07:12.395214 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.395187    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c0e3b3-6r9nz"
Nov 03 08:07:12.452747 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.452722    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb1f4c17-b03f-4c23-9742-1869aeb79837-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:12.452905 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.452752    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftncp\" (UniqueName: \"kubernetes.io/projected/bb1f4c17-b03f-4c23-9742-1869aeb79837-kube-api-access-ftncp\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:12.690072 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.689918    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2c0e3b3-6r9nz"]
Nov 03 08:07:12.949209 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.949130    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw"
Nov 03 08:07:12.949628 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.949133    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw" event={"ID":"bb1f4c17-b03f-4c23-9742-1869aeb79837","Type":"ContainerDied","Data":"112c172899b8716906d035bedfcc074d5578923d6b832cb0eeb0f0ecd0df1116"}
Nov 03 08:07:12.949628 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.949304    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="112c172899b8716906d035bedfcc074d5578923d6b832cb0eeb0f0ecd0df1116"
Nov 03 08:07:12.951113 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.951087    5070 generic.go:356] "Generic (PLEG): container finished" podID="a919ae58-9b50-4450-a523-1f64f89433c0" containerID="aa46d4b6d698a4e7b7cc4a5ad699e84723f812e4ae848bb11fbbd9f6d866d3a8" exitCode=0
Nov 03 08:07:12.951262 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.951196    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl" event={"ID":"a919ae58-9b50-4450-a523-1f64f89433c0","Type":"ContainerDied","Data":"aa46d4b6d698a4e7b7cc4a5ad699e84723f812e4ae848bb11fbbd9f6d866d3a8"}
Nov 03 08:07:12.952945 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:12.952922    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c0e3b3-6r9nz" event={"ID":"9b9ce1ef-4f78-4fdf-b846-e209df28cafe","Type":"ContainerStarted","Data":"201f9197fd78ccbe923eed2b87835412424a3e36a802f6ec7645c01b1922b2b6"}
Nov 03 08:07:13.219696 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:13.219608    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 08:07:13.219696 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:13.219677    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s" podUID="f76ed594-a720-4470-85da-02885a26dd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 08:07:13.562690 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:13.562619    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9"]
Nov 03 08:07:13.563383 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:13.563366    5070 cpu_manager.go:414] "containerMap: removing container" podUID="bb1f4c17-b03f-4c23-9742-1869aeb79837" containerName="revoke-keys"
Nov 03 08:07:13.563427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:13.563384    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb1f4c17-b03f-4c23-9742-1869aeb79837" containerName="revoke-keys"
Nov 03 08:07:13.563501 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:13.563492    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb1f4c17-b03f-4c23-9742-1869aeb79837" containerName="revoke-keys"
Nov 03 08:07:13.567291 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:13.567274    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9"
Nov 03 08:07:13.575484 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:13.575455    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9"]
Nov 03 08:07:13.700810 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:13.700781    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj9dx\" (UniqueName: \"kubernetes.io/projected/7ce8710b-6c3b-4d5a-9826-f77113dc6630-kube-api-access-tj9dx\") pod \"maas-api-revoke-sub-77229df9-46vf9\" (UID: \"7ce8710b-6c3b-4d5a-9826-f77113dc6630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9"
Nov 03 08:07:13.700971 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:13.700907    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ce8710b-6c3b-4d5a-9826-f77113dc6630-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77229df9-46vf9\" (UID: \"7ce8710b-6c3b-4d5a-9826-f77113dc6630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9"
Nov 03 08:07:13.701063 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:13.701045    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ce8710b-6c3b-4d5a-9826-f77113dc6630-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77229df9-46vf9\" (UID: \"7ce8710b-6c3b-4d5a-9826-f77113dc6630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9"
Nov 03 08:07:13.802415 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:13.802390    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj9dx\" (UniqueName: \"kubernetes.io/projected/7ce8710b-6c3b-4d5a-9826-f77113dc6630-kube-api-access-tj9dx\") pod \"maas-api-revoke-sub-77229df9-46vf9\" (UID: \"7ce8710b-6c3b-4d5a-9826-f77113dc6630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9"
Nov 03 08:07:13.802524 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:13.802452    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ce8710b-6c3b-4d5a-9826-f77113dc6630-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77229df9-46vf9\" (UID: \"7ce8710b-6c3b-4d5a-9826-f77113dc6630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9"
Nov 03 08:07:13.802524 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:13.802518    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ce8710b-6c3b-4d5a-9826-f77113dc6630-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77229df9-46vf9\" (UID: \"7ce8710b-6c3b-4d5a-9826-f77113dc6630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9"
Nov 03 08:07:13.803466 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:13.803434    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ce8710b-6c3b-4d5a-9826-f77113dc6630-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77229df9-46vf9\" (UID: \"7ce8710b-6c3b-4d5a-9826-f77113dc6630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9"
Nov 03 08:07:13.805292 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:13.805266    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ce8710b-6c3b-4d5a-9826-f77113dc6630-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77229df9-46vf9\" (UID: \"7ce8710b-6c3b-4d5a-9826-f77113dc6630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9"
Nov 03 08:07:13.809242 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:13.809220    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj9dx\" (UniqueName: \"kubernetes.io/projected/7ce8710b-6c3b-4d5a-9826-f77113dc6630-kube-api-access-tj9dx\") pod \"maas-api-revoke-sub-77229df9-46vf9\" (UID: \"7ce8710b-6c3b-4d5a-9826-f77113dc6630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9"
Nov 03 08:07:13.886792 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:13.886769    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9"
Nov 03 08:07:13.986025 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:13.984783    5070 generic.go:356] "Generic (PLEG): container finished" podID="9b9ce1ef-4f78-4fdf-b846-e209df28cafe" containerID="de26cf873275581845a1390e58363d09325fbc458ed82008aca6490eba57e0c4" exitCode=0
Nov 03 08:07:13.986025 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:13.985604    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c0e3b3-6r9nz" event={"ID":"9b9ce1ef-4f78-4fdf-b846-e209df28cafe","Type":"ContainerDied","Data":"de26cf873275581845a1390e58363d09325fbc458ed82008aca6490eba57e0c4"}
Nov 03 08:07:14.178131 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:14.178100    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9"]
Nov 03 08:07:14.219361 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:14.219328    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 08:07:14.219471 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:14.219377    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s" podUID="f76ed594-a720-4470-85da-02885a26dd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 08:07:14.268117 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:14.268088    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl"
Nov 03 08:07:14.414862 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:14.414834    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a919ae58-9b50-4450-a523-1f64f89433c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a919ae58-9b50-4450-a523-1f64f89433c0-maas-api-xks-ca\") pod \"a919ae58-9b50-4450-a523-1f64f89433c0\" (UID: \"a919ae58-9b50-4450-a523-1f64f89433c0\") "
Nov 03 08:07:14.415129 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:14.415113    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a919ae58-9b50-4450-a523-1f64f89433c0-kube-api-access-fd8c4\" (UniqueName: \"kubernetes.io/projected/a919ae58-9b50-4450-a523-1f64f89433c0-kube-api-access-fd8c4\") pod \"a919ae58-9b50-4450-a523-1f64f89433c0\" (UID: \"a919ae58-9b50-4450-a523-1f64f89433c0\") "
Nov 03 08:07:14.415219 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:14.415153    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a919ae58-9b50-4450-a523-1f64f89433c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a919ae58-9b50-4450-a523-1f64f89433c0-maas-api-openshift-ca\") pod \"a919ae58-9b50-4450-a523-1f64f89433c0\" (UID: \"a919ae58-9b50-4450-a523-1f64f89433c0\") "
Nov 03 08:07:14.415603 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:14.415571    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a919ae58-9b50-4450-a523-1f64f89433c0-maas-api-openshift-ca" pod "a919ae58-9b50-4450-a523-1f64f89433c0" (UID: "a919ae58-9b50-4450-a523-1f64f89433c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:07:14.415869 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:14.415845    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a919ae58-9b50-4450-a523-1f64f89433c0-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:14.417095 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:14.417071    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a919ae58-9b50-4450-a523-1f64f89433c0-maas-api-xks-ca" pod "a919ae58-9b50-4450-a523-1f64f89433c0" (UID: "a919ae58-9b50-4450-a523-1f64f89433c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:07:14.417328 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:14.417302    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a919ae58-9b50-4450-a523-1f64f89433c0-kube-api-access-fd8c4" pod "a919ae58-9b50-4450-a523-1f64f89433c0" (UID: "a919ae58-9b50-4450-a523-1f64f89433c0"). InnerVolumeSpecName "kube-api-access-fd8c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:07:14.521248 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:14.518808    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd8c4\" (UniqueName: \"kubernetes.io/projected/a919ae58-9b50-4450-a523-1f64f89433c0-kube-api-access-fd8c4\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:14.521248 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:14.518849    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a919ae58-9b50-4450-a523-1f64f89433c0-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:14.995465 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:14.995428    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9" event={"ID":"7ce8710b-6c3b-4d5a-9826-f77113dc6630","Type":"ContainerStarted","Data":"c8c3c7e1e424b062b9af045390c62e3017143b418d3d4659a5323c9fc992d70c"}
Nov 03 08:07:14.995935 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:14.995523    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9" event={"ID":"7ce8710b-6c3b-4d5a-9826-f77113dc6630","Type":"ContainerStarted","Data":"227547db79582d120e118c3cda933edf4baa01d67661bb4460103f39e750e049"}
Nov 03 08:07:14.996906 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:14.996878    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl"
Nov 03 08:07:14.997028 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:14.996911    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl" event={"ID":"a919ae58-9b50-4450-a523-1f64f89433c0","Type":"ContainerDied","Data":"a1cd9daac944e922eae8c1f67228b8d12e92992ea9dc96a3d05e9a046537f09c"}
Nov 03 08:07:14.997028 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:14.996948    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1cd9daac944e922eae8c1f67228b8d12e92992ea9dc96a3d05e9a046537f09c"
Nov 03 08:07:15.014941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:15.014898    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9" podStartSLOduration=2.014884546 podStartE2EDuration="2.014884546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:15.008218669 +0000 UTC m=+833.122682582" watchObservedRunningTime="2026-11-03 08:07:15.014884546 +0000 UTC m=+833.129348477"
Nov 03 08:07:15.159225 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:15.159194    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c0e3b3-6r9nz"
Nov 03 08:07:15.225648 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:15.225622    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:15.226550 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:15.226520    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:15.227638 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:15.227616    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:07:15.329856 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:15.329768    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b9ce1ef-4f78-4fdf-b846-e209df28cafe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b9ce1ef-4f78-4fdf-b846-e209df28cafe-maas-api-openshift-ca\") pod \"9b9ce1ef-4f78-4fdf-b846-e209df28cafe\" (UID: \"9b9ce1ef-4f78-4fdf-b846-e209df28cafe\") "
Nov 03 08:07:15.329856 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:15.329831    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b9ce1ef-4f78-4fdf-b846-e209df28cafe-kube-api-access-9lwjr\" (UniqueName: \"kubernetes.io/projected/9b9ce1ef-4f78-4fdf-b846-e209df28cafe-kube-api-access-9lwjr\") pod \"9b9ce1ef-4f78-4fdf-b846-e209df28cafe\" (UID: \"9b9ce1ef-4f78-4fdf-b846-e209df28cafe\") "
Nov 03 08:07:15.330070 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:15.329888    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b9ce1ef-4f78-4fdf-b846-e209df28cafe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b9ce1ef-4f78-4fdf-b846-e209df28cafe-maas-api-xks-ca\") pod \"9b9ce1ef-4f78-4fdf-b846-e209df28cafe\" (UID: \"9b9ce1ef-4f78-4fdf-b846-e209df28cafe\") "
Nov 03 08:07:15.330178 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:15.330147    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b9ce1ef-4f78-4fdf-b846-e209df28cafe-maas-api-openshift-ca" pod "9b9ce1ef-4f78-4fdf-b846-e209df28cafe" (UID: "9b9ce1ef-4f78-4fdf-b846-e209df28cafe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:07:15.331367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:15.331332    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b9ce1ef-4f78-4fdf-b846-e209df28cafe-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:15.332582 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:15.332553    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9ce1ef-4f78-4fdf-b846-e209df28cafe-kube-api-access-9lwjr" pod "9b9ce1ef-4f78-4fdf-b846-e209df28cafe" (UID: "9b9ce1ef-4f78-4fdf-b846-e209df28cafe"). InnerVolumeSpecName "kube-api-access-9lwjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:07:15.332679 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:15.332600    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b9ce1ef-4f78-4fdf-b846-e209df28cafe-maas-api-xks-ca" pod "9b9ce1ef-4f78-4fdf-b846-e209df28cafe" (UID: "9b9ce1ef-4f78-4fdf-b846-e209df28cafe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:07:15.433651 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:15.433611    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lwjr\" (UniqueName: \"kubernetes.io/projected/9b9ce1ef-4f78-4fdf-b846-e209df28cafe-kube-api-access-9lwjr\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:15.433651 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:15.433653    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b9ce1ef-4f78-4fdf-b846-e209df28cafe-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:16.007901 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:16.007864    5070 generic.go:356] "Generic (PLEG): container finished" podID="7ce8710b-6c3b-4d5a-9826-f77113dc6630" containerID="c8c3c7e1e424b062b9af045390c62e3017143b418d3d4659a5323c9fc992d70c" exitCode=0
Nov 03 08:07:16.008358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:16.007948    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9" event={"ID":"7ce8710b-6c3b-4d5a-9826-f77113dc6630","Type":"ContainerDied","Data":"c8c3c7e1e424b062b9af045390c62e3017143b418d3d4659a5323c9fc992d70c"}
Nov 03 08:07:16.009680 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:16.009661    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c0e3b3-6r9nz"
Nov 03 08:07:16.009834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:16.009702    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2c0e3b3-6r9nz" event={"ID":"9b9ce1ef-4f78-4fdf-b846-e209df28cafe","Type":"ContainerDied","Data":"201f9197fd78ccbe923eed2b87835412424a3e36a802f6ec7645c01b1922b2b6"}
Nov 03 08:07:16.009834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:16.009738    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="201f9197fd78ccbe923eed2b87835412424a3e36a802f6ec7645c01b1922b2b6"
Nov 03 08:07:17.195215 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:17.195189    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9"
Nov 03 08:07:17.280821 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:17.280737    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ce8710b-6c3b-4d5a-9826-f77113dc6630-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ce8710b-6c3b-4d5a-9826-f77113dc6630-maas-api-xks-ca\") pod \"7ce8710b-6c3b-4d5a-9826-f77113dc6630\" (UID: \"7ce8710b-6c3b-4d5a-9826-f77113dc6630\") "
Nov 03 08:07:17.280978 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:17.280960    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ce8710b-6c3b-4d5a-9826-f77113dc6630-kube-api-access-tj9dx\" (UniqueName: \"kubernetes.io/projected/7ce8710b-6c3b-4d5a-9826-f77113dc6630-kube-api-access-tj9dx\") pod \"7ce8710b-6c3b-4d5a-9826-f77113dc6630\" (UID: \"7ce8710b-6c3b-4d5a-9826-f77113dc6630\") "
Nov 03 08:07:17.281049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:17.281043    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ce8710b-6c3b-4d5a-9826-f77113dc6630-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ce8710b-6c3b-4d5a-9826-f77113dc6630-maas-api-openshift-ca\") pod \"7ce8710b-6c3b-4d5a-9826-f77113dc6630\" (UID: \"7ce8710b-6c3b-4d5a-9826-f77113dc6630\") "
Nov 03 08:07:17.281404 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:17.281371    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ce8710b-6c3b-4d5a-9826-f77113dc6630-maas-api-openshift-ca" pod "7ce8710b-6c3b-4d5a-9826-f77113dc6630" (UID: "7ce8710b-6c3b-4d5a-9826-f77113dc6630"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:07:17.281521 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:17.281512    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ce8710b-6c3b-4d5a-9826-f77113dc6630-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:17.282817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:17.282792    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ce8710b-6c3b-4d5a-9826-f77113dc6630-maas-api-xks-ca" pod "7ce8710b-6c3b-4d5a-9826-f77113dc6630" (UID: "7ce8710b-6c3b-4d5a-9826-f77113dc6630"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:07:17.283183 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:17.283163    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce8710b-6c3b-4d5a-9826-f77113dc6630-kube-api-access-tj9dx" pod "7ce8710b-6c3b-4d5a-9826-f77113dc6630" (UID: "7ce8710b-6c3b-4d5a-9826-f77113dc6630"). InnerVolumeSpecName "kube-api-access-tj9dx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:07:17.382343 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:17.382317    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj9dx\" (UniqueName: \"kubernetes.io/projected/7ce8710b-6c3b-4d5a-9826-f77113dc6630-kube-api-access-tj9dx\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:17.382343 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:17.382340    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ce8710b-6c3b-4d5a-9826-f77113dc6630-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:18.034317 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:18.034283    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9"
Nov 03 08:07:18.034478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:18.034325    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9" event={"ID":"7ce8710b-6c3b-4d5a-9826-f77113dc6630","Type":"ContainerDied","Data":"227547db79582d120e118c3cda933edf4baa01d67661bb4460103f39e750e049"}
Nov 03 08:07:18.034478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:18.034381    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="227547db79582d120e118c3cda933edf4baa01d67661bb4460103f39e750e049"
Nov 03 08:07:20.580547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.580505    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db487e8-kcmw4"]
Nov 03 08:07:20.581534 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.581508    5070 cpu_manager.go:414] "containerMap: removing container" podUID="a919ae58-9b50-4450-a523-1f64f89433c0" containerName="revoke-keys"
Nov 03 08:07:20.581645 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.581535    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a919ae58-9b50-4450-a523-1f64f89433c0" containerName="revoke-keys"
Nov 03 08:07:20.581645 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.581594    5070 cpu_manager.go:414] "containerMap: removing container" podUID="7ce8710b-6c3b-4d5a-9826-f77113dc6630" containerName="revoke-keys"
Nov 03 08:07:20.581645 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.581603    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ce8710b-6c3b-4d5a-9826-f77113dc6630" containerName="revoke-keys"
Nov 03 08:07:20.581645 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.581642    5070 cpu_manager.go:414] "containerMap: removing container" podUID="9b9ce1ef-4f78-4fdf-b846-e209df28cafe" containerName="revoke-keys"
Nov 03 08:07:20.581900 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.581650    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b9ce1ef-4f78-4fdf-b846-e209df28cafe" containerName="revoke-keys"
Nov 03 08:07:20.581900 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.581848    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b9ce1ef-4f78-4fdf-b846-e209df28cafe" containerName="revoke-keys"
Nov 03 08:07:20.581900 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.581870    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="a919ae58-9b50-4450-a523-1f64f89433c0" containerName="revoke-keys"
Nov 03 08:07:20.581900 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.581890    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ce8710b-6c3b-4d5a-9826-f77113dc6630" containerName="revoke-keys"
Nov 03 08:07:20.585717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.585521    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db487e8-kcmw4"
Nov 03 08:07:20.588694 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.588668    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:07:20.588853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.588715    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:07:20.588918 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.588882    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:07:20.588983 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.588940    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:07:20.592337 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.592312    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db487e8-kcmw4"]
Nov 03 08:07:20.627616 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.627588    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8hs7\" (UniqueName: \"kubernetes.io/projected/a769bcbf-c5e1-4a55-a47e-cc82372f982a-kube-api-access-l8hs7\") pod \"maas-api-revoke-sub-8db487e8-kcmw4\" (UID: \"a769bcbf-c5e1-4a55-a47e-cc82372f982a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db487e8-kcmw4"
Nov 03 08:07:20.627781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.627750    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a769bcbf-c5e1-4a55-a47e-cc82372f982a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8db487e8-kcmw4\" (UID: \"a769bcbf-c5e1-4a55-a47e-cc82372f982a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db487e8-kcmw4"
Nov 03 08:07:20.627899 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.627885    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a769bcbf-c5e1-4a55-a47e-cc82372f982a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8db487e8-kcmw4\" (UID: \"a769bcbf-c5e1-4a55-a47e-cc82372f982a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db487e8-kcmw4"
Nov 03 08:07:20.729487 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.729457    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a769bcbf-c5e1-4a55-a47e-cc82372f982a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8db487e8-kcmw4\" (UID: \"a769bcbf-c5e1-4a55-a47e-cc82372f982a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db487e8-kcmw4"
Nov 03 08:07:20.729649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.729600    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8hs7\" (UniqueName: \"kubernetes.io/projected/a769bcbf-c5e1-4a55-a47e-cc82372f982a-kube-api-access-l8hs7\") pod \"maas-api-revoke-sub-8db487e8-kcmw4\" (UID: \"a769bcbf-c5e1-4a55-a47e-cc82372f982a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db487e8-kcmw4"
Nov 03 08:07:20.729715 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.729673    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a769bcbf-c5e1-4a55-a47e-cc82372f982a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8db487e8-kcmw4\" (UID: \"a769bcbf-c5e1-4a55-a47e-cc82372f982a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db487e8-kcmw4"
Nov 03 08:07:20.730280 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.730257    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a769bcbf-c5e1-4a55-a47e-cc82372f982a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8db487e8-kcmw4\" (UID: \"a769bcbf-c5e1-4a55-a47e-cc82372f982a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db487e8-kcmw4"
Nov 03 08:07:20.731810 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.731787    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a769bcbf-c5e1-4a55-a47e-cc82372f982a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8db487e8-kcmw4\" (UID: \"a769bcbf-c5e1-4a55-a47e-cc82372f982a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db487e8-kcmw4"
Nov 03 08:07:20.736478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.736459    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8hs7\" (UniqueName: \"kubernetes.io/projected/a769bcbf-c5e1-4a55-a47e-cc82372f982a-kube-api-access-l8hs7\") pod \"maas-api-revoke-sub-8db487e8-kcmw4\" (UID: \"a769bcbf-c5e1-4a55-a47e-cc82372f982a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db487e8-kcmw4"
Nov 03 08:07:20.904957 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:20.904933    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db487e8-kcmw4"
Nov 03 08:07:21.192793 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:07:21.192743    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda769bcbf_c5e1_4a55_a47e_cc82372f982a.slice/crio-094f74a915219c192fae30f70301bd661bcc968d83b1eaa0487ce4ac9b5b8bcb WatchSource:0}: Error finding container 094f74a915219c192fae30f70301bd661bcc968d83b1eaa0487ce4ac9b5b8bcb: Status 404 returned error can't find the container with id 094f74a915219c192fae30f70301bd661bcc968d83b1eaa0487ce4ac9b5b8bcb
Nov 03 08:07:21.194518 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:21.194492    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db487e8-kcmw4"]
Nov 03 08:07:21.879831 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:21.879798    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g"]
Nov 03 08:07:21.884345 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:21.884318    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g"
Nov 03 08:07:21.891287 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:21.891264    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g"]
Nov 03 08:07:21.948573 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:21.948544    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9d86f14-d2d5-4166-a8ab-14db3536afa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f133ab31-tf49g\" (UID: \"a9d86f14-d2d5-4166-a8ab-14db3536afa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g"
Nov 03 08:07:21.948856 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:21.948834    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfzkk\" (UniqueName: \"kubernetes.io/projected/a9d86f14-d2d5-4166-a8ab-14db3536afa4-kube-api-access-mfzkk\") pod \"maas-api-revoke-sub-f133ab31-tf49g\" (UID: \"a9d86f14-d2d5-4166-a8ab-14db3536afa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g"
Nov 03 08:07:21.949018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:21.949002    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9d86f14-d2d5-4166-a8ab-14db3536afa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f133ab31-tf49g\" (UID: \"a9d86f14-d2d5-4166-a8ab-14db3536afa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g"
Nov 03 08:07:22.050344 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:22.050310    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfzkk\" (UniqueName: \"kubernetes.io/projected/a9d86f14-d2d5-4166-a8ab-14db3536afa4-kube-api-access-mfzkk\") pod \"maas-api-revoke-sub-f133ab31-tf49g\" (UID: \"a9d86f14-d2d5-4166-a8ab-14db3536afa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g"
Nov 03 08:07:22.050600 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:22.050439    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9d86f14-d2d5-4166-a8ab-14db3536afa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f133ab31-tf49g\" (UID: \"a9d86f14-d2d5-4166-a8ab-14db3536afa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g"
Nov 03 08:07:22.050600 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:22.050512    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9d86f14-d2d5-4166-a8ab-14db3536afa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f133ab31-tf49g\" (UID: \"a9d86f14-d2d5-4166-a8ab-14db3536afa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g"
Nov 03 08:07:22.051220 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:22.051197    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9d86f14-d2d5-4166-a8ab-14db3536afa4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f133ab31-tf49g\" (UID: \"a9d86f14-d2d5-4166-a8ab-14db3536afa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g"
Nov 03 08:07:22.052905 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:22.052878    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9d86f14-d2d5-4166-a8ab-14db3536afa4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f133ab31-tf49g\" (UID: \"a9d86f14-d2d5-4166-a8ab-14db3536afa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g"
Nov 03 08:07:22.057170 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:07:22.057142    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:07:22.058986 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:22.058957    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfzkk\" (UniqueName: \"kubernetes.io/projected/a9d86f14-d2d5-4166-a8ab-14db3536afa4-kube-api-access-mfzkk\") pod \"maas-api-revoke-sub-f133ab31-tf49g\" (UID: \"a9d86f14-d2d5-4166-a8ab-14db3536afa4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g"
Nov 03 08:07:22.076727 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:22.076703    5070 generic.go:356] "Generic (PLEG): container finished" podID="a769bcbf-c5e1-4a55-a47e-cc82372f982a" containerID="0719a67acecd83c256f950d076472b200421777355d3c17a05eb7fc98619970c" exitCode=0
Nov 03 08:07:22.076836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:22.076790    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db487e8-kcmw4" event={"ID":"a769bcbf-c5e1-4a55-a47e-cc82372f982a","Type":"ContainerDied","Data":"0719a67acecd83c256f950d076472b200421777355d3c17a05eb7fc98619970c"}
Nov 03 08:07:22.076886 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:22.076846    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db487e8-kcmw4" event={"ID":"a769bcbf-c5e1-4a55-a47e-cc82372f982a","Type":"ContainerStarted","Data":"094f74a915219c192fae30f70301bd661bcc968d83b1eaa0487ce4ac9b5b8bcb"}
Nov 03 08:07:22.206253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:22.206196    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g"
Nov 03 08:07:22.496708 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:07:22.496677    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d86f14_d2d5_4166_a8ab_14db3536afa4.slice/crio-cd4e092c08cfa987236f22dbbead5abf287e7cb66995d339cbd8cab90be04c6f WatchSource:0}: Error finding container cd4e092c08cfa987236f22dbbead5abf287e7cb66995d339cbd8cab90be04c6f: Status 404 returned error can't find the container with id cd4e092c08cfa987236f22dbbead5abf287e7cb66995d339cbd8cab90be04c6f
Nov 03 08:07:22.498317 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:22.498269    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g"]
Nov 03 08:07:23.087638 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.087548    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g" event={"ID":"a9d86f14-d2d5-4166-a8ab-14db3536afa4","Type":"ContainerStarted","Data":"a2d5421c7810e4ae75b249e62a1a6fc0a56ce16723cb1ec69a4d05bdb3ca2b36"}
Nov 03 08:07:23.087638 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.087600    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g" event={"ID":"a9d86f14-d2d5-4166-a8ab-14db3536afa4","Type":"ContainerStarted","Data":"cd4e092c08cfa987236f22dbbead5abf287e7cb66995d339cbd8cab90be04c6f"}
Nov 03 08:07:23.105545 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.105489    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g" podStartSLOduration=2.105473403 podStartE2EDuration="2.105473403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:23.100315607 +0000 UTC m=+841.214779538" watchObservedRunningTime="2026-11-03 08:07:23.105473403 +0000 UTC m=+841.219937335"
Nov 03 08:07:23.304394 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.304363    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db487e8-kcmw4"
Nov 03 08:07:23.376856 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.376752    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a769bcbf-c5e1-4a55-a47e-cc82372f982a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a769bcbf-c5e1-4a55-a47e-cc82372f982a-maas-api-xks-ca\") pod \"a769bcbf-c5e1-4a55-a47e-cc82372f982a\" (UID: \"a769bcbf-c5e1-4a55-a47e-cc82372f982a\") "
Nov 03 08:07:23.377062 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.376997    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a769bcbf-c5e1-4a55-a47e-cc82372f982a-kube-api-access-l8hs7\" (UniqueName: \"kubernetes.io/projected/a769bcbf-c5e1-4a55-a47e-cc82372f982a-kube-api-access-l8hs7\") pod \"a769bcbf-c5e1-4a55-a47e-cc82372f982a\" (UID: \"a769bcbf-c5e1-4a55-a47e-cc82372f982a\") "
Nov 03 08:07:23.377062 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.377044    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a769bcbf-c5e1-4a55-a47e-cc82372f982a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a769bcbf-c5e1-4a55-a47e-cc82372f982a-maas-api-openshift-ca\") pod \"a769bcbf-c5e1-4a55-a47e-cc82372f982a\" (UID: \"a769bcbf-c5e1-4a55-a47e-cc82372f982a\") "
Nov 03 08:07:23.379080 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.379053    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a769bcbf-c5e1-4a55-a47e-cc82372f982a-maas-api-openshift-ca" pod "a769bcbf-c5e1-4a55-a47e-cc82372f982a" (UID: "a769bcbf-c5e1-4a55-a47e-cc82372f982a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:07:23.380950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.380914    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a769bcbf-c5e1-4a55-a47e-cc82372f982a-maas-api-xks-ca" pod "a769bcbf-c5e1-4a55-a47e-cc82372f982a" (UID: "a769bcbf-c5e1-4a55-a47e-cc82372f982a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:07:23.381496 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.381459    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a769bcbf-c5e1-4a55-a47e-cc82372f982a-kube-api-access-l8hs7" pod "a769bcbf-c5e1-4a55-a47e-cc82372f982a" (UID: "a769bcbf-c5e1-4a55-a47e-cc82372f982a"). InnerVolumeSpecName "kube-api-access-l8hs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:07:23.478962 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.478938    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a769bcbf-c5e1-4a55-a47e-cc82372f982a-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:23.478962 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.478963    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8hs7\" (UniqueName: \"kubernetes.io/projected/a769bcbf-c5e1-4a55-a47e-cc82372f982a-kube-api-access-l8hs7\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:23.479086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.478973    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a769bcbf-c5e1-4a55-a47e-cc82372f982a-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:23.800647 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.800619    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp"]
Nov 03 08:07:23.801738 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.801719    5070 cpu_manager.go:414] "containerMap: removing container" podUID="a769bcbf-c5e1-4a55-a47e-cc82372f982a" containerName="revoke-keys"
Nov 03 08:07:23.801815 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.801741    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a769bcbf-c5e1-4a55-a47e-cc82372f982a" containerName="revoke-keys"
Nov 03 08:07:23.801897 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.801886    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="a769bcbf-c5e1-4a55-a47e-cc82372f982a" containerName="revoke-keys"
Nov 03 08:07:23.805671 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.805654    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp"
Nov 03 08:07:23.814098 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.814073    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp"]
Nov 03 08:07:23.883401 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.883373    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27583ffd-7959-4158-a5c2-f523349cda90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b7103d0-9jzfp\" (UID: \"27583ffd-7959-4158-a5c2-f523349cda90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp"
Nov 03 08:07:23.883583 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.883562    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kpzf\" (UniqueName: \"kubernetes.io/projected/27583ffd-7959-4158-a5c2-f523349cda90-kube-api-access-6kpzf\") pod \"maas-api-revoke-sub-3b7103d0-9jzfp\" (UID: \"27583ffd-7959-4158-a5c2-f523349cda90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp"
Nov 03 08:07:23.883657 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.883602    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27583ffd-7959-4158-a5c2-f523349cda90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b7103d0-9jzfp\" (UID: \"27583ffd-7959-4158-a5c2-f523349cda90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp"
Nov 03 08:07:23.984513 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.984483    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27583ffd-7959-4158-a5c2-f523349cda90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b7103d0-9jzfp\" (UID: \"27583ffd-7959-4158-a5c2-f523349cda90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp"
Nov 03 08:07:23.984651 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.984586    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kpzf\" (UniqueName: \"kubernetes.io/projected/27583ffd-7959-4158-a5c2-f523349cda90-kube-api-access-6kpzf\") pod \"maas-api-revoke-sub-3b7103d0-9jzfp\" (UID: \"27583ffd-7959-4158-a5c2-f523349cda90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp"
Nov 03 08:07:23.984651 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.984610    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27583ffd-7959-4158-a5c2-f523349cda90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b7103d0-9jzfp\" (UID: \"27583ffd-7959-4158-a5c2-f523349cda90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp"
Nov 03 08:07:23.985207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.985183    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27583ffd-7959-4158-a5c2-f523349cda90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b7103d0-9jzfp\" (UID: \"27583ffd-7959-4158-a5c2-f523349cda90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp"
Nov 03 08:07:23.986924 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.986904    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27583ffd-7959-4158-a5c2-f523349cda90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b7103d0-9jzfp\" (UID: \"27583ffd-7959-4158-a5c2-f523349cda90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp"
Nov 03 08:07:23.991541 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:23.991520    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kpzf\" (UniqueName: \"kubernetes.io/projected/27583ffd-7959-4158-a5c2-f523349cda90-kube-api-access-6kpzf\") pod \"maas-api-revoke-sub-3b7103d0-9jzfp\" (UID: \"27583ffd-7959-4158-a5c2-f523349cda90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp"
Nov 03 08:07:24.098749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:24.098717    5070 generic.go:356] "Generic (PLEG): container finished" podID="a9d86f14-d2d5-4166-a8ab-14db3536afa4" containerID="a2d5421c7810e4ae75b249e62a1a6fc0a56ce16723cb1ec69a4d05bdb3ca2b36" exitCode=0
Nov 03 08:07:24.099119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:24.098793    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g" event={"ID":"a9d86f14-d2d5-4166-a8ab-14db3536afa4","Type":"ContainerDied","Data":"a2d5421c7810e4ae75b249e62a1a6fc0a56ce16723cb1ec69a4d05bdb3ca2b36"}
Nov 03 08:07:24.100595 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:24.100564    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db487e8-kcmw4" event={"ID":"a769bcbf-c5e1-4a55-a47e-cc82372f982a","Type":"ContainerDied","Data":"094f74a915219c192fae30f70301bd661bcc968d83b1eaa0487ce4ac9b5b8bcb"}
Nov 03 08:07:24.100595 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:24.100578    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8db487e8-kcmw4"
Nov 03 08:07:24.100786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:24.100602    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="094f74a915219c192fae30f70301bd661bcc968d83b1eaa0487ce4ac9b5b8bcb"
Nov 03 08:07:24.123588 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:24.123563    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp"
Nov 03 08:07:24.413596 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:07:24.413561    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27583ffd_7959_4158_a5c2_f523349cda90.slice/crio-3678667b9d920eeca64da50504e5937a5cad99a6297ba37d3f948f30fcca461f WatchSource:0}: Error finding container 3678667b9d920eeca64da50504e5937a5cad99a6297ba37d3f948f30fcca461f: Status 404 returned error can't find the container with id 3678667b9d920eeca64da50504e5937a5cad99a6297ba37d3f948f30fcca461f
Nov 03 08:07:24.415522 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:24.415467    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp"]
Nov 03 08:07:25.111101 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.111066    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp" event={"ID":"27583ffd-7959-4158-a5c2-f523349cda90","Type":"ContainerStarted","Data":"f777a77aa63ca4bb1fe3a81b77eeef2d45346bc68e85a2f01a5255c0bccaf3eb"}
Nov 03 08:07:25.111564 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.111120    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp" event={"ID":"27583ffd-7959-4158-a5c2-f523349cda90","Type":"ContainerStarted","Data":"3678667b9d920eeca64da50504e5937a5cad99a6297ba37d3f948f30fcca461f"}
Nov 03 08:07:25.132030 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.131976    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp" podStartSLOduration=2.1319609330000002 podStartE2EDuration="2.131960933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:25.128974976 +0000 UTC m=+843.243439011" watchObservedRunningTime="2026-11-03 08:07:25.131960933 +0000 UTC m=+843.246424866"
Nov 03 08:07:25.399411 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.399386    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g"
Nov 03 08:07:25.514708 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.514670    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9d86f14-d2d5-4166-a8ab-14db3536afa4-kube-api-access-mfzkk\" (UniqueName: \"kubernetes.io/projected/a9d86f14-d2d5-4166-a8ab-14db3536afa4-kube-api-access-mfzkk\") pod \"a9d86f14-d2d5-4166-a8ab-14db3536afa4\" (UID: \"a9d86f14-d2d5-4166-a8ab-14db3536afa4\") "
Nov 03 08:07:25.514914 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.514810    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9d86f14-d2d5-4166-a8ab-14db3536afa4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9d86f14-d2d5-4166-a8ab-14db3536afa4-maas-api-openshift-ca\") pod \"a9d86f14-d2d5-4166-a8ab-14db3536afa4\" (UID: \"a9d86f14-d2d5-4166-a8ab-14db3536afa4\") "
Nov 03 08:07:25.514914 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.514829    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9d86f14-d2d5-4166-a8ab-14db3536afa4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9d86f14-d2d5-4166-a8ab-14db3536afa4-maas-api-xks-ca\") pod \"a9d86f14-d2d5-4166-a8ab-14db3536afa4\" (UID: \"a9d86f14-d2d5-4166-a8ab-14db3536afa4\") "
Nov 03 08:07:25.515167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.515140    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d86f14-d2d5-4166-a8ab-14db3536afa4-maas-api-openshift-ca" pod "a9d86f14-d2d5-4166-a8ab-14db3536afa4" (UID: "a9d86f14-d2d5-4166-a8ab-14db3536afa4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:07:25.515579 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.515560    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9d86f14-d2d5-4166-a8ab-14db3536afa4-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:25.517030 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.516994    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9d86f14-d2d5-4166-a8ab-14db3536afa4-maas-api-xks-ca" pod "a9d86f14-d2d5-4166-a8ab-14db3536afa4" (UID: "a9d86f14-d2d5-4166-a8ab-14db3536afa4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:07:25.517030 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.517002    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d86f14-d2d5-4166-a8ab-14db3536afa4-kube-api-access-mfzkk" pod "a9d86f14-d2d5-4166-a8ab-14db3536afa4" (UID: "a9d86f14-d2d5-4166-a8ab-14db3536afa4"). InnerVolumeSpecName "kube-api-access-mfzkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:07:25.603894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.603867    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d4e3763-7j8cd"]
Nov 03 08:07:25.604578 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.604558    5070 cpu_manager.go:414] "containerMap: removing container" podUID="a9d86f14-d2d5-4166-a8ab-14db3536afa4" containerName="revoke-keys"
Nov 03 08:07:25.604674 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.604581    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9d86f14-d2d5-4166-a8ab-14db3536afa4" containerName="revoke-keys"
Nov 03 08:07:25.604832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.604818    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9d86f14-d2d5-4166-a8ab-14db3536afa4" containerName="revoke-keys"
Nov 03 08:07:25.608853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.608835    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d4e3763-7j8cd"
Nov 03 08:07:25.616406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.616382    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d4e3763-7j8cd"]
Nov 03 08:07:25.617060 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.617034    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfzkk\" (UniqueName: \"kubernetes.io/projected/a9d86f14-d2d5-4166-a8ab-14db3536afa4-kube-api-access-mfzkk\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:25.617155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.617065    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9d86f14-d2d5-4166-a8ab-14db3536afa4-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:25.718825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.718796    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f364abcd-f24f-493e-9962-d20498f7abcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d4e3763-7j8cd\" (UID: \"f364abcd-f24f-493e-9962-d20498f7abcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d4e3763-7j8cd"
Nov 03 08:07:25.718951 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.718839    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f364abcd-f24f-493e-9962-d20498f7abcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d4e3763-7j8cd\" (UID: \"f364abcd-f24f-493e-9962-d20498f7abcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d4e3763-7j8cd"
Nov 03 08:07:25.719004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.718947    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q65r\" (UniqueName: \"kubernetes.io/projected/f364abcd-f24f-493e-9962-d20498f7abcc-kube-api-access-4q65r\") pod \"maas-api-revoke-sub-1d4e3763-7j8cd\" (UID: \"f364abcd-f24f-493e-9962-d20498f7abcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d4e3763-7j8cd"
Nov 03 08:07:25.734734 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.734701    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d326211f-bk7nw"]
Nov 03 08:07:25.739414 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.739386    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d326211f-bk7nw"
Nov 03 08:07:25.747952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.747928    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d326211f-bk7nw"]
Nov 03 08:07:25.820816 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.820790    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f364abcd-f24f-493e-9962-d20498f7abcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d4e3763-7j8cd\" (UID: \"f364abcd-f24f-493e-9962-d20498f7abcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d4e3763-7j8cd"
Nov 03 08:07:25.820941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.820838    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f364abcd-f24f-493e-9962-d20498f7abcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d4e3763-7j8cd\" (UID: \"f364abcd-f24f-493e-9962-d20498f7abcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d4e3763-7j8cd"
Nov 03 08:07:25.820941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.820890    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bbb013d-3d06-48e6-8656-b86171b2edbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d326211f-bk7nw\" (UID: \"5bbb013d-3d06-48e6-8656-b86171b2edbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d326211f-bk7nw"
Nov 03 08:07:25.820941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.820908    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj689\" (UniqueName: \"kubernetes.io/projected/5bbb013d-3d06-48e6-8656-b86171b2edbf-kube-api-access-wj689\") pod \"maas-api-revoke-sub-d326211f-bk7nw\" (UID: \"5bbb013d-3d06-48e6-8656-b86171b2edbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d326211f-bk7nw"
Nov 03 08:07:25.821048 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.821017    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q65r\" (UniqueName: \"kubernetes.io/projected/f364abcd-f24f-493e-9962-d20498f7abcc-kube-api-access-4q65r\") pod \"maas-api-revoke-sub-1d4e3763-7j8cd\" (UID: \"f364abcd-f24f-493e-9962-d20498f7abcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d4e3763-7j8cd"
Nov 03 08:07:25.821048 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.821039    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bbb013d-3d06-48e6-8656-b86171b2edbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d326211f-bk7nw\" (UID: \"5bbb013d-3d06-48e6-8656-b86171b2edbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d326211f-bk7nw"
Nov 03 08:07:25.821595 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.821575    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f364abcd-f24f-493e-9962-d20498f7abcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d4e3763-7j8cd\" (UID: \"f364abcd-f24f-493e-9962-d20498f7abcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d4e3763-7j8cd"
Nov 03 08:07:25.823178 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.823155    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f364abcd-f24f-493e-9962-d20498f7abcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d4e3763-7j8cd\" (UID: \"f364abcd-f24f-493e-9962-d20498f7abcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d4e3763-7j8cd"
Nov 03 08:07:25.827745 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.827723    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q65r\" (UniqueName: \"kubernetes.io/projected/f364abcd-f24f-493e-9962-d20498f7abcc-kube-api-access-4q65r\") pod \"maas-api-revoke-sub-1d4e3763-7j8cd\" (UID: \"f364abcd-f24f-493e-9962-d20498f7abcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d4e3763-7j8cd"
Nov 03 08:07:25.922686 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.922620    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bbb013d-3d06-48e6-8656-b86171b2edbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d326211f-bk7nw\" (UID: \"5bbb013d-3d06-48e6-8656-b86171b2edbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d326211f-bk7nw"
Nov 03 08:07:25.922686 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.922654    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj689\" (UniqueName: \"kubernetes.io/projected/5bbb013d-3d06-48e6-8656-b86171b2edbf-kube-api-access-wj689\") pod \"maas-api-revoke-sub-d326211f-bk7nw\" (UID: \"5bbb013d-3d06-48e6-8656-b86171b2edbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d326211f-bk7nw"
Nov 03 08:07:25.922853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.922707    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bbb013d-3d06-48e6-8656-b86171b2edbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d326211f-bk7nw\" (UID: \"5bbb013d-3d06-48e6-8656-b86171b2edbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d326211f-bk7nw"
Nov 03 08:07:25.923389 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.923366    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bbb013d-3d06-48e6-8656-b86171b2edbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d326211f-bk7nw\" (UID: \"5bbb013d-3d06-48e6-8656-b86171b2edbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d326211f-bk7nw"
Nov 03 08:07:25.924990 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.924962    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bbb013d-3d06-48e6-8656-b86171b2edbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d326211f-bk7nw\" (UID: \"5bbb013d-3d06-48e6-8656-b86171b2edbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d326211f-bk7nw"
Nov 03 08:07:25.929085 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.929065    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj689\" (UniqueName: \"kubernetes.io/projected/5bbb013d-3d06-48e6-8656-b86171b2edbf-kube-api-access-wj689\") pod \"maas-api-revoke-sub-d326211f-bk7nw\" (UID: \"5bbb013d-3d06-48e6-8656-b86171b2edbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d326211f-bk7nw"
Nov 03 08:07:25.934863 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:25.934847    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d4e3763-7j8cd"
Nov 03 08:07:26.066908 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:26.066878    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d326211f-bk7nw"
Nov 03 08:07:26.123463 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:26.123431    5070 generic.go:356] "Generic (PLEG): container finished" podID="27583ffd-7959-4158-a5c2-f523349cda90" containerID="f777a77aa63ca4bb1fe3a81b77eeef2d45346bc68e85a2f01a5255c0bccaf3eb" exitCode=0
Nov 03 08:07:26.123906 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:26.123515    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp" event={"ID":"27583ffd-7959-4158-a5c2-f523349cda90","Type":"ContainerDied","Data":"f777a77aa63ca4bb1fe3a81b77eeef2d45346bc68e85a2f01a5255c0bccaf3eb"}
Nov 03 08:07:26.125454 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:26.125431    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g" event={"ID":"a9d86f14-d2d5-4166-a8ab-14db3536afa4","Type":"ContainerDied","Data":"cd4e092c08cfa987236f22dbbead5abf287e7cb66995d339cbd8cab90be04c6f"}
Nov 03 08:07:26.125564 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:26.125469    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd4e092c08cfa987236f22dbbead5abf287e7cb66995d339cbd8cab90be04c6f"
Nov 03 08:07:26.125564 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:26.125514    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g"
Nov 03 08:07:26.226743 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:26.226715    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d4e3763-7j8cd"]
Nov 03 08:07:26.359894 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:07:26.359861    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bbb013d_3d06_48e6_8656_b86171b2edbf.slice/crio-a5375b9ae0c3a26a374cd4f2949a53ac742251eecff8c06613735c2a788a9a5d WatchSource:0}: Error finding container a5375b9ae0c3a26a374cd4f2949a53ac742251eecff8c06613735c2a788a9a5d: Status 404 returned error can't find the container with id a5375b9ae0c3a26a374cd4f2949a53ac742251eecff8c06613735c2a788a9a5d
Nov 03 08:07:26.361827 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:26.361805    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d326211f-bk7nw"]
Nov 03 08:07:27.179000 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:27.178902    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d4e3763-7j8cd" event={"ID":"f364abcd-f24f-493e-9962-d20498f7abcc","Type":"ContainerStarted","Data":"14e5028ae6881956d235957782fb0f2748b9b8364c34b7d9b18121b91540878d"}
Nov 03 08:07:27.181530 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:27.181499    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d326211f-bk7nw" event={"ID":"5bbb013d-3d06-48e6-8656-b86171b2edbf","Type":"ContainerStarted","Data":"a5375b9ae0c3a26a374cd4f2949a53ac742251eecff8c06613735c2a788a9a5d"}
Nov 03 08:07:27.600537 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:27.599270    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:27.914618 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:27.913897    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:28.116106 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:28.114848    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:28.241749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:28.227841    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:28.264616 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:28.262148    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:07:28.264616 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:28.263179    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:07:28.339810 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:28.334939    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:28.364809 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:28.350855    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:28.364809 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:28.363295    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:28.528842 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:28.507468    5070 generic.go:356] "Generic (PLEG): container finished" podID="f364abcd-f24f-493e-9962-d20498f7abcc" containerID="240e1d97c6b26eb6ae368e594992aaed38e832fed1d5f47a3b604056638d7c36" exitCode=0
Nov 03 08:07:28.528842 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:28.507883    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d4e3763-7j8cd" event={"ID":"f364abcd-f24f-493e-9962-d20498f7abcc","Type":"ContainerDied","Data":"240e1d97c6b26eb6ae368e594992aaed38e832fed1d5f47a3b604056638d7c36"}
Nov 03 08:07:28.907550 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:28.893067    5070 generic.go:356] "Generic (PLEG): container finished" podID="5bbb013d-3d06-48e6-8656-b86171b2edbf" containerID="16c0392664a0395133f1a3c7b4da5f0db6d46141930d1bd32f0b1b3e562e712d" exitCode=0
Nov 03 08:07:28.907550 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:28.893211    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d326211f-bk7nw" event={"ID":"5bbb013d-3d06-48e6-8656-b86171b2edbf","Type":"ContainerDied","Data":"16c0392664a0395133f1a3c7b4da5f0db6d46141930d1bd32f0b1b3e562e712d"}
Nov 03 08:07:28.907550 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:28.907057    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:07:28.934856 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:28.908526    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:07:28.963165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:28.939141    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:28.976930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:28.965506    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:07:29.181844 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:29.170123    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:07:29.275873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:29.182115    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded"
Nov 03 08:07:29.275873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:29.230695    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:29.275873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:29.243905    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:07:29.275873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:29.243967    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:07:29.278078 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:29.276664    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:29.279355 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:29.278264    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:29.303240 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:29.279939    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:07:29.303240 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:29.280000    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:07:29.406800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:29.344113    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:07:29.406800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:29.344440    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:07:29.406800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:29.356143    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:29.406800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:29.362813    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:07:29.406800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:29.368637    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:29.406800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:29.373963    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:07:29.900793 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:29.870318    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": context deadline exceeded"
Nov 03 08:07:30.198514 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:29.929851    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:30.335207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:29.945130    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:30.345129 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:30.247899    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:30.369787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:30.247938    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:07:30.553842 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:30.292563    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:30.716624 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:30.292604    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:30.717198 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:30.359324    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:07:30.760933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:30.359375    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:07:30.761633 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:30.379618    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:07:30.762043 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:30.696611    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp"
Nov 03 08:07:30.859835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:30.843035    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded"
Nov 03 08:07:30.910800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:30.903004    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp"
Nov 03 08:07:30.932497 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:30.906345    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp" event={"ID":"27583ffd-7959-4158-a5c2-f523349cda90","Type":"ContainerDied","Data":"3678667b9d920eeca64da50504e5937a5cad99a6297ba37d3f948f30fcca461f"}
Nov 03 08:07:30.933455 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:30.933427    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3678667b9d920eeca64da50504e5937a5cad99a6297ba37d3f948f30fcca461f"
Nov 03 08:07:31.093784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:31.052443    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27583ffd-7959-4158-a5c2-f523349cda90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27583ffd-7959-4158-a5c2-f523349cda90-maas-api-xks-ca\") pod \"27583ffd-7959-4158-a5c2-f523349cda90\" (UID: \"27583ffd-7959-4158-a5c2-f523349cda90\") "
Nov 03 08:07:31.217092 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:31.217004    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27583ffd-7959-4158-a5c2-f523349cda90-kube-api-access-6kpzf\" (UniqueName: \"kubernetes.io/projected/27583ffd-7959-4158-a5c2-f523349cda90-kube-api-access-6kpzf\") pod \"27583ffd-7959-4158-a5c2-f523349cda90\" (UID: \"27583ffd-7959-4158-a5c2-f523349cda90\") "
Nov 03 08:07:31.224788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:31.221121    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27583ffd-7959-4158-a5c2-f523349cda90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27583ffd-7959-4158-a5c2-f523349cda90-maas-api-openshift-ca\") pod \"27583ffd-7959-4158-a5c2-f523349cda90\" (UID: \"27583ffd-7959-4158-a5c2-f523349cda90\") "
Nov 03 08:07:31.319111 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:31.305866    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27583ffd-7959-4158-a5c2-f523349cda90-maas-api-openshift-ca" pod "27583ffd-7959-4158-a5c2-f523349cda90" (UID: "27583ffd-7959-4158-a5c2-f523349cda90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:07:31.341607 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:31.337844    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:31.347086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:31.347038    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:31.800832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:31.739982    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:31.927783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:31.740338    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27583ffd-7959-4158-a5c2-f523349cda90-maas-api-xks-ca" pod "27583ffd-7959-4158-a5c2-f523349cda90" (UID: "27583ffd-7959-4158-a5c2-f523349cda90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:07:31.948782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:31.940878    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27583ffd-7959-4158-a5c2-f523349cda90-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:31.995932 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:31.995888    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27583ffd-7959-4158-a5c2-f523349cda90-kube-api-access-6kpzf" pod "27583ffd-7959-4158-a5c2-f523349cda90" (UID: "27583ffd-7959-4158-a5c2-f523349cda90"). InnerVolumeSpecName "kube-api-access-6kpzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:07:32.082839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:32.051391    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:07:32.321781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:32.294005    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:07:32.827107 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:32.366093    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:32.827107 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:32.825825    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr" podUID="64c976ca-8c6d-4835-b5ca-ba88be627ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:32.827107 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:32.441373    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:32.827107 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:32.441068    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:07:32.827107 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:32.823133    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j" podUID="67eb2350-8895-4e7d-b1f2-e78e1ff5cf23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:32.833949 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:32.832143    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:32.833949 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:32.832867    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:07:33.016946 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.009032    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27583ffd-7959-4158-a5c2-f523349cda90-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:33.063447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.063414    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kpzf\" (UniqueName: \"kubernetes.io/projected/27583ffd-7959-4158-a5c2-f523349cda90-kube-api-access-6kpzf\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:33.066821 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.019076    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:33.085954 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.085863    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:33.203058 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.201405    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded"
Nov 03 08:07:33.203058 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.201827    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:07:33.203058 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.202069    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:07:33.256141 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.256099    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:07:33.258592 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.258556    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d4e3763-7j8cd"
Nov 03 08:07:33.259095 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.259075    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:07:33.264165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.264147    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d326211f-bk7nw"
Nov 03 08:07:33.265053 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.265029    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:07:33.271518 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.270054    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:07:33.272115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.272084    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:07:33.273236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.273215    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:07:33.278708 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.278166    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:07:33.286788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.286769    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:07:33.288490 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.288467    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:07:33.299892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.299870    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"]
Nov 03 08:07:33.300578 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.300563    5070 cpu_manager.go:414] "containerMap: removing container" podUID="27583ffd-7959-4158-a5c2-f523349cda90" containerName="revoke-keys"
Nov 03 08:07:33.300617 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.300579    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27583ffd-7959-4158-a5c2-f523349cda90" containerName="revoke-keys"
Nov 03 08:07:33.300617 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.300593    5070 cpu_manager.go:414] "containerMap: removing container" podUID="f364abcd-f24f-493e-9962-d20498f7abcc" containerName="revoke-keys"
Nov 03 08:07:33.300617 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.300598    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f364abcd-f24f-493e-9962-d20498f7abcc" containerName="revoke-keys"
Nov 03 08:07:33.300770 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.300633    5070 cpu_manager.go:414] "containerMap: removing container" podUID="5bbb013d-3d06-48e6-8656-b86171b2edbf" containerName="revoke-keys"
Nov 03 08:07:33.300770 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.300638    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bbb013d-3d06-48e6-8656-b86171b2edbf" containerName="revoke-keys"
Nov 03 08:07:33.300770 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.300737    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bbb013d-3d06-48e6-8656-b86171b2edbf" containerName="revoke-keys"
Nov 03 08:07:33.300770 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.300749    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="27583ffd-7959-4158-a5c2-f523349cda90" containerName="revoke-keys"
Nov 03 08:07:33.300889 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.300810    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="f364abcd-f24f-493e-9962-d20498f7abcc" containerName="revoke-keys"
Nov 03 08:07:33.304462 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.304447    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.304802 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.304783    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"]
Nov 03 08:07:33.404164 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.404130    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f364abcd-f24f-493e-9962-d20498f7abcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f364abcd-f24f-493e-9962-d20498f7abcc-maas-api-xks-ca\") pod \"f364abcd-f24f-493e-9962-d20498f7abcc\" (UID: \"f364abcd-f24f-493e-9962-d20498f7abcc\") "
Nov 03 08:07:33.404403 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.404271    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f364abcd-f24f-493e-9962-d20498f7abcc-kube-api-access-4q65r\" (UniqueName: \"kubernetes.io/projected/f364abcd-f24f-493e-9962-d20498f7abcc-kube-api-access-4q65r\") pod \"f364abcd-f24f-493e-9962-d20498f7abcc\" (UID: \"f364abcd-f24f-493e-9962-d20498f7abcc\") "
Nov 03 08:07:33.404490 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.404430    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bbb013d-3d06-48e6-8656-b86171b2edbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bbb013d-3d06-48e6-8656-b86171b2edbf-maas-api-openshift-ca\") pod \"5bbb013d-3d06-48e6-8656-b86171b2edbf\" (UID: \"5bbb013d-3d06-48e6-8656-b86171b2edbf\") "
Nov 03 08:07:33.404490 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.404471    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bbb013d-3d06-48e6-8656-b86171b2edbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bbb013d-3d06-48e6-8656-b86171b2edbf-maas-api-xks-ca\") pod \"5bbb013d-3d06-48e6-8656-b86171b2edbf\" (UID: \"5bbb013d-3d06-48e6-8656-b86171b2edbf\") "
Nov 03 08:07:33.404636 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.404619    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f364abcd-f24f-493e-9962-d20498f7abcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f364abcd-f24f-493e-9962-d20498f7abcc-maas-api-openshift-ca\") pod \"f364abcd-f24f-493e-9962-d20498f7abcc\" (UID: \"f364abcd-f24f-493e-9962-d20498f7abcc\") "
Nov 03 08:07:33.404674 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.404656    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bbb013d-3d06-48e6-8656-b86171b2edbf-kube-api-access-wj689\" (UniqueName: \"kubernetes.io/projected/5bbb013d-3d06-48e6-8656-b86171b2edbf-kube-api-access-wj689\") pod \"5bbb013d-3d06-48e6-8656-b86171b2edbf\" (UID: \"5bbb013d-3d06-48e6-8656-b86171b2edbf\") "
Nov 03 08:07:33.404976 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.404948    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bbb013d-3d06-48e6-8656-b86171b2edbf-maas-api-openshift-ca" pod "5bbb013d-3d06-48e6-8656-b86171b2edbf" (UID: "5bbb013d-3d06-48e6-8656-b86171b2edbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:07:33.405054 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.405032    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.405156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.405140    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.405221 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.405200    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.405372 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.405343    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.405880 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.405857    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.413817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.406189    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f364abcd-f24f-493e-9962-d20498f7abcc-maas-api-openshift-ca" pod "f364abcd-f24f-493e-9962-d20498f7abcc" (UID: "f364abcd-f24f-493e-9962-d20498f7abcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:07:33.413817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.407921    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.413817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.407964    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.413817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.408897    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.413817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.408930    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb9nr\" (UniqueName: \"kubernetes.io/projected/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-kube-api-access-hb9nr\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.413817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.409160    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bbb013d-3d06-48e6-8656-b86171b2edbf-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:33.413817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.409190    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f364abcd-f24f-493e-9962-d20498f7abcc-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:33.413817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.410813    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bbb013d-3d06-48e6-8656-b86171b2edbf-kube-api-access-wj689" pod "5bbb013d-3d06-48e6-8656-b86171b2edbf" (UID: "5bbb013d-3d06-48e6-8656-b86171b2edbf"). InnerVolumeSpecName "kube-api-access-wj689". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:07:33.415880 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.415075    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f364abcd-f24f-493e-9962-d20498f7abcc-maas-api-xks-ca" pod "f364abcd-f24f-493e-9962-d20498f7abcc" (UID: "f364abcd-f24f-493e-9962-d20498f7abcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:07:33.415880 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.415172    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f364abcd-f24f-493e-9962-d20498f7abcc-kube-api-access-4q65r" pod "f364abcd-f24f-493e-9962-d20498f7abcc" (UID: "f364abcd-f24f-493e-9962-d20498f7abcc"). InnerVolumeSpecName "kube-api-access-4q65r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:07:33.416740 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.416708    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bbb013d-3d06-48e6-8656-b86171b2edbf-maas-api-xks-ca" pod "5bbb013d-3d06-48e6-8656-b86171b2edbf" (UID: "5bbb013d-3d06-48e6-8656-b86171b2edbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:07:33.511014 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.510982    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.511186 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.511041    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.511398 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.511369    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-data\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.511499 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.511406    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.511499 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.511461    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.511499 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.511478    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb9nr\" (UniqueName: \"kubernetes.io/projected/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-kube-api-access-hb9nr\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.511666 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.511515    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.511666 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.511549    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.511666 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.511574    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.511666 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.511615    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.511901 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.511695    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj689\" (UniqueName: \"kubernetes.io/projected/5bbb013d-3d06-48e6-8656-b86171b2edbf-kube-api-access-wj689\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:33.511901 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.511706    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f364abcd-f24f-493e-9962-d20498f7abcc-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:33.511901 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.511715    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q65r\" (UniqueName: \"kubernetes.io/projected/f364abcd-f24f-493e-9962-d20498f7abcc-kube-api-access-4q65r\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:33.511901 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.511725    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bbb013d-3d06-48e6-8656-b86171b2edbf-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:33.511901 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.511785    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istiod-ca-cert\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.512144 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.512052    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-workload-certs\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.512199 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.512158    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-workload-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.512366 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.512316    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-credential-socket\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.514652 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.514623    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-envoy\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.514746 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.514690    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-podinfo\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.526319 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.526294    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-token\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.527328 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.527307    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb9nr\" (UniqueName: \"kubernetes.io/projected/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-kube-api-access-hb9nr\") pod \"e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") " pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.635002 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.634953    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:33.932815 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:07:33.932779    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e6709c_fa81_4a1c_81ee_e6cbecd5c7c4.slice/crio-85f88274ad26b4db6ba41d1ac50db83853cccc43825e53df2f7edb1cbbb2706a WatchSource:0}: Error finding container 85f88274ad26b4db6ba41d1ac50db83853cccc43825e53df2f7edb1cbbb2706a: Status 404 returned error can't find the container with id 85f88274ad26b4db6ba41d1ac50db83853cccc43825e53df2f7edb1cbbb2706a
Nov 03 08:07:33.933744 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.933718    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"]
Nov 03 08:07:33.935546 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.935509    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:07:33.935629 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.935574    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:07:33.935629 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:33.935610    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:07:34.259207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:34.259115    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7" event={"ID":"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4","Type":"ContainerStarted","Data":"0fa8ebd3a1da0d6692fcdb1427352c97a6148ae8aefc7656bf0c48b4f692ef15"}
Nov 03 08:07:34.259207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:34.259178    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7" event={"ID":"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4","Type":"ContainerStarted","Data":"85f88274ad26b4db6ba41d1ac50db83853cccc43825e53df2f7edb1cbbb2706a"}
Nov 03 08:07:34.260655 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:34.260630    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d326211f-bk7nw"
Nov 03 08:07:34.260811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:34.260652    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d326211f-bk7nw" event={"ID":"5bbb013d-3d06-48e6-8656-b86171b2edbf","Type":"ContainerDied","Data":"a5375b9ae0c3a26a374cd4f2949a53ac742251eecff8c06613735c2a788a9a5d"}
Nov 03 08:07:34.260811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:34.260698    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5375b9ae0c3a26a374cd4f2949a53ac742251eecff8c06613735c2a788a9a5d"
Nov 03 08:07:34.263102 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:34.263061    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d4e3763-7j8cd" event={"ID":"f364abcd-f24f-493e-9962-d20498f7abcc","Type":"ContainerDied","Data":"14e5028ae6881956d235957782fb0f2748b9b8364c34b7d9b18121b91540878d"}
Nov 03 08:07:34.263213 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:34.263127    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14e5028ae6881956d235957782fb0f2748b9b8364c34b7d9b18121b91540878d"
Nov 03 08:07:34.263213 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:34.263086    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d4e3763-7j8cd"
Nov 03 08:07:34.280205 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:34.280158    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7" podStartSLOduration=4.280146999 podStartE2EDuration="4.280146999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:34.274740536 +0000 UTC m=+852.389204467" watchObservedRunningTime="2026-11-03 08:07:34.280146999 +0000 UTC m=+852.394610929"
Nov 03 08:07:34.635455 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:34.635428    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:35.636892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:35.636858    5070 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 03 08:07:35.637343 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:35.636921    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7" podUID="a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 03 08:07:36.400352 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.400315    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cf09e14-nm7h8"]
Nov 03 08:07:36.405359 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.405332    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf09e14-nm7h8"
Nov 03 08:07:36.408738 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.408649    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:07:36.408926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.408901    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:07:36.409049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.408953    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:07:36.409049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.408990    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:07:36.415594 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.415068    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cf09e14-nm7h8"]
Nov 03 08:07:36.450996 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.450968    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6"]
Nov 03 08:07:36.490693 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.490662    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6"]
Nov 03 08:07:36.490913 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.490766    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6"
Nov 03 08:07:36.509165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.509132    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjctn\" (UniqueName: \"kubernetes.io/projected/3d7b0574-49d7-4b35-94cb-065d525d881a-kube-api-access-wjctn\") pod \"maas-api-revoke-sub-7cf09e14-nm7h8\" (UID: \"3d7b0574-49d7-4b35-94cb-065d525d881a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf09e14-nm7h8"
Nov 03 08:07:36.509286 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.509191    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7b0574-49d7-4b35-94cb-065d525d881a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cf09e14-nm7h8\" (UID: \"3d7b0574-49d7-4b35-94cb-065d525d881a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf09e14-nm7h8"
Nov 03 08:07:36.509354 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.509337    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7b0574-49d7-4b35-94cb-065d525d881a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cf09e14-nm7h8\" (UID: \"3d7b0574-49d7-4b35-94cb-065d525d881a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf09e14-nm7h8"
Nov 03 08:07:36.612623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.612578    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2blj5\" (UniqueName: \"kubernetes.io/projected/3acd647f-bec3-4ace-8674-2c7009f9e91c-kube-api-access-2blj5\") pod \"maas-api-revoke-sub-f48a026a-8nzm6\" (UID: \"3acd647f-bec3-4ace-8674-2c7009f9e91c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6"
Nov 03 08:07:36.612890 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.612650    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjctn\" (UniqueName: \"kubernetes.io/projected/3d7b0574-49d7-4b35-94cb-065d525d881a-kube-api-access-wjctn\") pod \"maas-api-revoke-sub-7cf09e14-nm7h8\" (UID: \"3d7b0574-49d7-4b35-94cb-065d525d881a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf09e14-nm7h8"
Nov 03 08:07:36.612890 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.612709    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7b0574-49d7-4b35-94cb-065d525d881a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cf09e14-nm7h8\" (UID: \"3d7b0574-49d7-4b35-94cb-065d525d881a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf09e14-nm7h8"
Nov 03 08:07:36.612890 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.612862    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7b0574-49d7-4b35-94cb-065d525d881a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cf09e14-nm7h8\" (UID: \"3d7b0574-49d7-4b35-94cb-065d525d881a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf09e14-nm7h8"
Nov 03 08:07:36.613100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.612923    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3acd647f-bec3-4ace-8674-2c7009f9e91c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f48a026a-8nzm6\" (UID: \"3acd647f-bec3-4ace-8674-2c7009f9e91c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6"
Nov 03 08:07:36.613100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.613025    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3acd647f-bec3-4ace-8674-2c7009f9e91c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f48a026a-8nzm6\" (UID: \"3acd647f-bec3-4ace-8674-2c7009f9e91c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6"
Nov 03 08:07:36.614157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.613922    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7b0574-49d7-4b35-94cb-065d525d881a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cf09e14-nm7h8\" (UID: \"3d7b0574-49d7-4b35-94cb-065d525d881a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf09e14-nm7h8"
Nov 03 08:07:36.621905 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.618076    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7b0574-49d7-4b35-94cb-065d525d881a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cf09e14-nm7h8\" (UID: \"3d7b0574-49d7-4b35-94cb-065d525d881a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf09e14-nm7h8"
Nov 03 08:07:36.625734 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.625031    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjctn\" (UniqueName: \"kubernetes.io/projected/3d7b0574-49d7-4b35-94cb-065d525d881a-kube-api-access-wjctn\") pod \"maas-api-revoke-sub-7cf09e14-nm7h8\" (UID: \"3d7b0574-49d7-4b35-94cb-065d525d881a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf09e14-nm7h8"
Nov 03 08:07:36.635600 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.635574    5070 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 03 08:07:36.635735 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.635625    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7" podUID="a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 03 08:07:36.715134 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.715056    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3acd647f-bec3-4ace-8674-2c7009f9e91c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f48a026a-8nzm6\" (UID: \"3acd647f-bec3-4ace-8674-2c7009f9e91c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6"
Nov 03 08:07:36.715537 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.715213    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2blj5\" (UniqueName: \"kubernetes.io/projected/3acd647f-bec3-4ace-8674-2c7009f9e91c-kube-api-access-2blj5\") pod \"maas-api-revoke-sub-f48a026a-8nzm6\" (UID: \"3acd647f-bec3-4ace-8674-2c7009f9e91c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6"
Nov 03 08:07:36.715537 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.715391    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3acd647f-bec3-4ace-8674-2c7009f9e91c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f48a026a-8nzm6\" (UID: \"3acd647f-bec3-4ace-8674-2c7009f9e91c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6"
Nov 03 08:07:36.716081 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.716060    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3acd647f-bec3-4ace-8674-2c7009f9e91c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f48a026a-8nzm6\" (UID: \"3acd647f-bec3-4ace-8674-2c7009f9e91c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6"
Nov 03 08:07:36.717426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.717404    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3acd647f-bec3-4ace-8674-2c7009f9e91c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f48a026a-8nzm6\" (UID: \"3acd647f-bec3-4ace-8674-2c7009f9e91c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6"
Nov 03 08:07:36.722347 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.722318    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2blj5\" (UniqueName: \"kubernetes.io/projected/3acd647f-bec3-4ace-8674-2c7009f9e91c-kube-api-access-2blj5\") pod \"maas-api-revoke-sub-f48a026a-8nzm6\" (UID: \"3acd647f-bec3-4ace-8674-2c7009f9e91c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6"
Nov 03 08:07:36.728889 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.728870    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf09e14-nm7h8"
Nov 03 08:07:36.811865 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:36.811544    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6"
Nov 03 08:07:37.083926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:37.083901    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cf09e14-nm7h8"]
Nov 03 08:07:37.155110 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:07:37.155086    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3acd647f_bec3_4ace_8674_2c7009f9e91c.slice/crio-145f2f7211b9a7de9c8ff0dec054965610f1774ef970f3daa63cc5fa304a9bdb WatchSource:0}: Error finding container 145f2f7211b9a7de9c8ff0dec054965610f1774ef970f3daa63cc5fa304a9bdb: Status 404 returned error can't find the container with id 145f2f7211b9a7de9c8ff0dec054965610f1774ef970f3daa63cc5fa304a9bdb
Nov 03 08:07:37.182776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:37.181807    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6"]
Nov 03 08:07:37.314283 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:37.314214    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf09e14-nm7h8" event={"ID":"3d7b0574-49d7-4b35-94cb-065d525d881a","Type":"ContainerStarted","Data":"776870b4b1026b824663b343deb53b69c2e4184d7165ff4c703ebf83cd57c6c8"}
Nov 03 08:07:37.315519 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:37.315489    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6" event={"ID":"3acd647f-bec3-4ace-8674-2c7009f9e91c","Type":"ContainerStarted","Data":"145f2f7211b9a7de9c8ff0dec054965610f1774ef970f3daa63cc5fa304a9bdb"}
Nov 03 08:07:37.636110 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:37.636081    5070 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 03 08:07:37.636253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:37.636136    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7" podUID="a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 03 08:07:38.255386 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:38.255359    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-425b2345-ctlh2"]
Nov 03 08:07:38.339109 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:38.338345    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-425b2345-ctlh2"]
Nov 03 08:07:38.339109 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:38.338468    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425b2345-ctlh2"
Nov 03 08:07:38.346631 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:38.346598    5070 generic.go:356] "Generic (PLEG): container finished" podID="3d7b0574-49d7-4b35-94cb-065d525d881a" containerID="551d089071bdf3e3e3a2da21b68c36436a882bb045fdc5a5974cfdca88dc1174" exitCode=0
Nov 03 08:07:38.346845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:38.346718    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf09e14-nm7h8" event={"ID":"3d7b0574-49d7-4b35-94cb-065d525d881a","Type":"ContainerDied","Data":"551d089071bdf3e3e3a2da21b68c36436a882bb045fdc5a5974cfdca88dc1174"}
Nov 03 08:07:38.350640 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:38.350579    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6" event={"ID":"3acd647f-bec3-4ace-8674-2c7009f9e91c","Type":"ContainerStarted","Data":"15c0b1253b5930b6f2cd7a09b6088dd5d384ff98edb7572fc526400c717b7d20"}
Nov 03 08:07:38.438748 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:38.438623    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6" podStartSLOduration=2.438605317 podStartE2EDuration="2.438605317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:38.436714415 +0000 UTC m=+856.551178349" watchObservedRunningTime="2026-11-03 08:07:38.438605317 +0000 UTC m=+856.553069249"
Nov 03 08:07:38.447069 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:38.447039    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204a59ad-3b92-4bc7-ad14-2dc39aa8a104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-425b2345-ctlh2\" (UID: \"204a59ad-3b92-4bc7-ad14-2dc39aa8a104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425b2345-ctlh2"
Nov 03 08:07:38.447244 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:38.447102    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204a59ad-3b92-4bc7-ad14-2dc39aa8a104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-425b2345-ctlh2\" (UID: \"204a59ad-3b92-4bc7-ad14-2dc39aa8a104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425b2345-ctlh2"
Nov 03 08:07:38.447345 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:38.447313    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4x22\" (UniqueName: \"kubernetes.io/projected/204a59ad-3b92-4bc7-ad14-2dc39aa8a104-kube-api-access-r4x22\") pod \"maas-api-revoke-sub-425b2345-ctlh2\" (UID: \"204a59ad-3b92-4bc7-ad14-2dc39aa8a104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425b2345-ctlh2"
Nov 03 08:07:38.549039 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:38.549000    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4x22\" (UniqueName: \"kubernetes.io/projected/204a59ad-3b92-4bc7-ad14-2dc39aa8a104-kube-api-access-r4x22\") pod \"maas-api-revoke-sub-425b2345-ctlh2\" (UID: \"204a59ad-3b92-4bc7-ad14-2dc39aa8a104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425b2345-ctlh2"
Nov 03 08:07:38.549283 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:38.549265    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204a59ad-3b92-4bc7-ad14-2dc39aa8a104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-425b2345-ctlh2\" (UID: \"204a59ad-3b92-4bc7-ad14-2dc39aa8a104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425b2345-ctlh2"
Nov 03 08:07:38.549359 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:38.549318    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204a59ad-3b92-4bc7-ad14-2dc39aa8a104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-425b2345-ctlh2\" (UID: \"204a59ad-3b92-4bc7-ad14-2dc39aa8a104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425b2345-ctlh2"
Nov 03 08:07:38.550050 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:38.550026    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204a59ad-3b92-4bc7-ad14-2dc39aa8a104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-425b2345-ctlh2\" (UID: \"204a59ad-3b92-4bc7-ad14-2dc39aa8a104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425b2345-ctlh2"
Nov 03 08:07:38.551709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:38.551683    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204a59ad-3b92-4bc7-ad14-2dc39aa8a104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-425b2345-ctlh2\" (UID: \"204a59ad-3b92-4bc7-ad14-2dc39aa8a104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425b2345-ctlh2"
Nov 03 08:07:38.591579 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:38.591545    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4x22\" (UniqueName: \"kubernetes.io/projected/204a59ad-3b92-4bc7-ad14-2dc39aa8a104-kube-api-access-r4x22\") pod \"maas-api-revoke-sub-425b2345-ctlh2\" (UID: \"204a59ad-3b92-4bc7-ad14-2dc39aa8a104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-425b2345-ctlh2"
Nov 03 08:07:38.638670 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:38.637063    5070 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 03 08:07:38.638670 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:38.637120    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7" podUID="a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 03 08:07:38.677130 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:38.677101    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425b2345-ctlh2"
Nov 03 08:07:39.002802 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:07:38.999931    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod204a59ad_3b92_4bc7_ad14_2dc39aa8a104.slice/crio-cf908c437c8063beb42a31f21904426e8069dd25b08693f85acb829a338e745c WatchSource:0}: Error finding container cf908c437c8063beb42a31f21904426e8069dd25b08693f85acb829a338e745c: Status 404 returned error can't find the container with id cf908c437c8063beb42a31f21904426e8069dd25b08693f85acb829a338e745c
Nov 03 08:07:39.002802 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.002785    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-425b2345-ctlh2"]
Nov 03 08:07:39.397642 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.397605    5070 generic.go:356] "Generic (PLEG): container finished" podID="3acd647f-bec3-4ace-8674-2c7009f9e91c" containerID="15c0b1253b5930b6f2cd7a09b6088dd5d384ff98edb7572fc526400c717b7d20" exitCode=0
Nov 03 08:07:39.399661 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.399000    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6" event={"ID":"3acd647f-bec3-4ace-8674-2c7009f9e91c","Type":"ContainerDied","Data":"15c0b1253b5930b6f2cd7a09b6088dd5d384ff98edb7572fc526400c717b7d20"}
Nov 03 08:07:39.403310 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.403285    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425b2345-ctlh2" event={"ID":"204a59ad-3b92-4bc7-ad14-2dc39aa8a104","Type":"ContainerStarted","Data":"cf908c437c8063beb42a31f21904426e8069dd25b08693f85acb829a338e745c"}
Nov 03 08:07:39.501989 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.500138    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3db6d499-75l5p"]
Nov 03 08:07:39.511585 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.511553    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3db6d499-75l5p"]
Nov 03 08:07:39.511845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.511649    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db6d499-75l5p"
Nov 03 08:07:39.595618 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.595577    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/588587e7-5d9a-41cf-baa4-1d998c96a47e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3db6d499-75l5p\" (UID: \"588587e7-5d9a-41cf-baa4-1d998c96a47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db6d499-75l5p"
Nov 03 08:07:39.596270 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.595744    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hxkn\" (UniqueName: \"kubernetes.io/projected/588587e7-5d9a-41cf-baa4-1d998c96a47e-kube-api-access-2hxkn\") pod \"maas-api-revoke-sub-3db6d499-75l5p\" (UID: \"588587e7-5d9a-41cf-baa4-1d998c96a47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db6d499-75l5p"
Nov 03 08:07:39.596270 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.595934    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/588587e7-5d9a-41cf-baa4-1d998c96a47e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3db6d499-75l5p\" (UID: \"588587e7-5d9a-41cf-baa4-1d998c96a47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db6d499-75l5p"
Nov 03 08:07:39.635651 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.635620    5070 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 03 08:07:39.635921 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.635890    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7" podUID="a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 03 08:07:39.697652 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.697623    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/588587e7-5d9a-41cf-baa4-1d998c96a47e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3db6d499-75l5p\" (UID: \"588587e7-5d9a-41cf-baa4-1d998c96a47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db6d499-75l5p"
Nov 03 08:07:39.697899 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.697878    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/588587e7-5d9a-41cf-baa4-1d998c96a47e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3db6d499-75l5p\" (UID: \"588587e7-5d9a-41cf-baa4-1d998c96a47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db6d499-75l5p"
Nov 03 08:07:39.698010 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.697994    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hxkn\" (UniqueName: \"kubernetes.io/projected/588587e7-5d9a-41cf-baa4-1d998c96a47e-kube-api-access-2hxkn\") pod \"maas-api-revoke-sub-3db6d499-75l5p\" (UID: \"588587e7-5d9a-41cf-baa4-1d998c96a47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db6d499-75l5p"
Nov 03 08:07:39.698938 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.698917    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/588587e7-5d9a-41cf-baa4-1d998c96a47e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3db6d499-75l5p\" (UID: \"588587e7-5d9a-41cf-baa4-1d998c96a47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db6d499-75l5p"
Nov 03 08:07:39.700026 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.700004    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/588587e7-5d9a-41cf-baa4-1d998c96a47e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3db6d499-75l5p\" (UID: \"588587e7-5d9a-41cf-baa4-1d998c96a47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db6d499-75l5p"
Nov 03 08:07:39.705086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.704910    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hxkn\" (UniqueName: \"kubernetes.io/projected/588587e7-5d9a-41cf-baa4-1d998c96a47e-kube-api-access-2hxkn\") pod \"maas-api-revoke-sub-3db6d499-75l5p\" (UID: \"588587e7-5d9a-41cf-baa4-1d998c96a47e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db6d499-75l5p"
Nov 03 08:07:39.737611 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.737589    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf09e14-nm7h8"
Nov 03 08:07:39.799405 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.799372    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d7b0574-49d7-4b35-94cb-065d525d881a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7b0574-49d7-4b35-94cb-065d525d881a-maas-api-openshift-ca\") pod \"3d7b0574-49d7-4b35-94cb-065d525d881a\" (UID: \"3d7b0574-49d7-4b35-94cb-065d525d881a\") "
Nov 03 08:07:39.799709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.799689    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d7b0574-49d7-4b35-94cb-065d525d881a-maas-api-openshift-ca" pod "3d7b0574-49d7-4b35-94cb-065d525d881a" (UID: "3d7b0574-49d7-4b35-94cb-065d525d881a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:07:39.799964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.799950    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d7b0574-49d7-4b35-94cb-065d525d881a-kube-api-access-wjctn\" (UniqueName: \"kubernetes.io/projected/3d7b0574-49d7-4b35-94cb-065d525d881a-kube-api-access-wjctn\") pod \"3d7b0574-49d7-4b35-94cb-065d525d881a\" (UID: \"3d7b0574-49d7-4b35-94cb-065d525d881a\") "
Nov 03 08:07:39.800376 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.800347    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d7b0574-49d7-4b35-94cb-065d525d881a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7b0574-49d7-4b35-94cb-065d525d881a-maas-api-xks-ca\") pod \"3d7b0574-49d7-4b35-94cb-065d525d881a\" (UID: \"3d7b0574-49d7-4b35-94cb-065d525d881a\") "
Nov 03 08:07:39.801289 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.801268    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d7b0574-49d7-4b35-94cb-065d525d881a-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:39.802485 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.802456    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7b0574-49d7-4b35-94cb-065d525d881a-kube-api-access-wjctn" pod "3d7b0574-49d7-4b35-94cb-065d525d881a" (UID: "3d7b0574-49d7-4b35-94cb-065d525d881a"). InnerVolumeSpecName "kube-api-access-wjctn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:07:39.803035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.803009    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d7b0574-49d7-4b35-94cb-065d525d881a-maas-api-xks-ca" pod "3d7b0574-49d7-4b35-94cb-065d525d881a" (UID: "3d7b0574-49d7-4b35-94cb-065d525d881a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:07:39.829831 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.829800    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db6d499-75l5p"
Nov 03 08:07:39.903052 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.903018    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjctn\" (UniqueName: \"kubernetes.io/projected/3d7b0574-49d7-4b35-94cb-065d525d881a-kube-api-access-wjctn\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:39.903052 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:39.903051    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d7b0574-49d7-4b35-94cb-065d525d881a-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:40.151366 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:07:40.151340    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588587e7_5d9a_41cf_baa4_1d998c96a47e.slice/crio-fe9b8ffa996334a11e40afc821941de53725bd5ab9b316ee6503b229f2059e69 WatchSource:0}: Error finding container fe9b8ffa996334a11e40afc821941de53725bd5ab9b316ee6503b229f2059e69: Status 404 returned error can't find the container with id fe9b8ffa996334a11e40afc821941de53725bd5ab9b316ee6503b229f2059e69
Nov 03 08:07:40.156419 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:40.156389    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3db6d499-75l5p"]
Nov 03 08:07:40.416244 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:40.416211    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db6d499-75l5p" event={"ID":"588587e7-5d9a-41cf-baa4-1d998c96a47e","Type":"ContainerStarted","Data":"fe9b8ffa996334a11e40afc821941de53725bd5ab9b316ee6503b229f2059e69"}
Nov 03 08:07:40.418049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:40.418021    5070 generic.go:356] "Generic (PLEG): container finished" podID="204a59ad-3b92-4bc7-ad14-2dc39aa8a104" containerID="c1b6d966650e3e097ce2a8b435c86ecd14a9cd040431c5b8828678f5f834b870" exitCode=0
Nov 03 08:07:40.418152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:40.418103    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425b2345-ctlh2" event={"ID":"204a59ad-3b92-4bc7-ad14-2dc39aa8a104","Type":"ContainerDied","Data":"c1b6d966650e3e097ce2a8b435c86ecd14a9cd040431c5b8828678f5f834b870"}
Nov 03 08:07:40.419641 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:40.419625    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf09e14-nm7h8"
Nov 03 08:07:40.419741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:40.419660    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cf09e14-nm7h8" event={"ID":"3d7b0574-49d7-4b35-94cb-065d525d881a","Type":"ContainerDied","Data":"776870b4b1026b824663b343deb53b69c2e4184d7165ff4c703ebf83cd57c6c8"}
Nov 03 08:07:40.419741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:40.419691    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="776870b4b1026b824663b343deb53b69c2e4184d7165ff4c703ebf83cd57c6c8"
Nov 03 08:07:40.625306 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:40.625221    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6"
Nov 03 08:07:40.636189 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:40.636157    5070 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Nov 03 08:07:40.636584 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:40.636537    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7" podUID="a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Nov 03 08:07:40.719213 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:40.719181    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3acd647f-bec3-4ace-8674-2c7009f9e91c-kube-api-access-2blj5\" (UniqueName: \"kubernetes.io/projected/3acd647f-bec3-4ace-8674-2c7009f9e91c-kube-api-access-2blj5\") pod \"3acd647f-bec3-4ace-8674-2c7009f9e91c\" (UID: \"3acd647f-bec3-4ace-8674-2c7009f9e91c\") "
Nov 03 08:07:40.719565 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:40.719545    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3acd647f-bec3-4ace-8674-2c7009f9e91c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3acd647f-bec3-4ace-8674-2c7009f9e91c-maas-api-xks-ca\") pod \"3acd647f-bec3-4ace-8674-2c7009f9e91c\" (UID: \"3acd647f-bec3-4ace-8674-2c7009f9e91c\") "
Nov 03 08:07:40.719970 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:40.719944    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3acd647f-bec3-4ace-8674-2c7009f9e91c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3acd647f-bec3-4ace-8674-2c7009f9e91c-maas-api-openshift-ca\") pod \"3acd647f-bec3-4ace-8674-2c7009f9e91c\" (UID: \"3acd647f-bec3-4ace-8674-2c7009f9e91c\") "
Nov 03 08:07:40.721146 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:40.721120    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3acd647f-bec3-4ace-8674-2c7009f9e91c-maas-api-openshift-ca" pod "3acd647f-bec3-4ace-8674-2c7009f9e91c" (UID: "3acd647f-bec3-4ace-8674-2c7009f9e91c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:07:40.733482 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:40.733459    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3acd647f-bec3-4ace-8674-2c7009f9e91c-maas-api-xks-ca" pod "3acd647f-bec3-4ace-8674-2c7009f9e91c" (UID: "3acd647f-bec3-4ace-8674-2c7009f9e91c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:07:40.733984 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:40.733961    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3acd647f-bec3-4ace-8674-2c7009f9e91c-kube-api-access-2blj5" pod "3acd647f-bec3-4ace-8674-2c7009f9e91c" (UID: "3acd647f-bec3-4ace-8674-2c7009f9e91c"). InnerVolumeSpecName "kube-api-access-2blj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:07:40.822131 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:40.822105    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2blj5\" (UniqueName: \"kubernetes.io/projected/3acd647f-bec3-4ace-8674-2c7009f9e91c-kube-api-access-2blj5\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:40.822131 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:40.822131    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3acd647f-bec3-4ace-8674-2c7009f9e91c-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:40.822288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:40.822141    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3acd647f-bec3-4ace-8674-2c7009f9e91c-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:41.434321 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:41.434290    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6"
Nov 03 08:07:41.434805 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:41.434289    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6" event={"ID":"3acd647f-bec3-4ace-8674-2c7009f9e91c","Type":"ContainerDied","Data":"145f2f7211b9a7de9c8ff0dec054965610f1774ef970f3daa63cc5fa304a9bdb"}
Nov 03 08:07:41.434805 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:41.434436    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="145f2f7211b9a7de9c8ff0dec054965610f1774ef970f3daa63cc5fa304a9bdb"
Nov 03 08:07:41.436671 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:41.436642    5070 generic.go:356] "Generic (PLEG): container finished" podID="588587e7-5d9a-41cf-baa4-1d998c96a47e" containerID="00f2351acc831976a876ea110f1c822dc134cb870bd18625b375bdabae409fcb" exitCode=0
Nov 03 08:07:41.436903 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:41.436869    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db6d499-75l5p" event={"ID":"588587e7-5d9a-41cf-baa4-1d998c96a47e","Type":"ContainerDied","Data":"00f2351acc831976a876ea110f1c822dc134cb870bd18625b375bdabae409fcb"}
Nov 03 08:07:41.642318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:41.642294    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:41.642781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:41.642743    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:41.643797 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:41.643774    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:07:41.687092 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:41.687002    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425b2345-ctlh2"
Nov 03 08:07:41.839505 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:41.839465    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/204a59ad-3b92-4bc7-ad14-2dc39aa8a104-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204a59ad-3b92-4bc7-ad14-2dc39aa8a104-maas-api-xks-ca\") pod \"204a59ad-3b92-4bc7-ad14-2dc39aa8a104\" (UID: \"204a59ad-3b92-4bc7-ad14-2dc39aa8a104\") "
Nov 03 08:07:41.839844 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:41.839821    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204a59ad-3b92-4bc7-ad14-2dc39aa8a104-kube-api-access-r4x22\" (UniqueName: \"kubernetes.io/projected/204a59ad-3b92-4bc7-ad14-2dc39aa8a104-kube-api-access-r4x22\") pod \"204a59ad-3b92-4bc7-ad14-2dc39aa8a104\" (UID: \"204a59ad-3b92-4bc7-ad14-2dc39aa8a104\") "
Nov 03 08:07:41.839957 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:41.839876    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204a59ad-3b92-4bc7-ad14-2dc39aa8a104-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204a59ad-3b92-4bc7-ad14-2dc39aa8a104-maas-api-openshift-ca\") pod \"204a59ad-3b92-4bc7-ad14-2dc39aa8a104\" (UID: \"204a59ad-3b92-4bc7-ad14-2dc39aa8a104\") "
Nov 03 08:07:41.843603 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:41.843571    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204a59ad-3b92-4bc7-ad14-2dc39aa8a104-maas-api-openshift-ca" pod "204a59ad-3b92-4bc7-ad14-2dc39aa8a104" (UID: "204a59ad-3b92-4bc7-ad14-2dc39aa8a104"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:07:41.846920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:41.846890    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/204a59ad-3b92-4bc7-ad14-2dc39aa8a104-maas-api-xks-ca" pod "204a59ad-3b92-4bc7-ad14-2dc39aa8a104" (UID: "204a59ad-3b92-4bc7-ad14-2dc39aa8a104"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:07:41.848770 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:41.848728    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204a59ad-3b92-4bc7-ad14-2dc39aa8a104-kube-api-access-r4x22" pod "204a59ad-3b92-4bc7-ad14-2dc39aa8a104" (UID: "204a59ad-3b92-4bc7-ad14-2dc39aa8a104"). InnerVolumeSpecName "kube-api-access-r4x22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:07:41.944922 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:41.944832    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4x22\" (UniqueName: \"kubernetes.io/projected/204a59ad-3b92-4bc7-ad14-2dc39aa8a104-kube-api-access-r4x22\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:41.944922 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:41.944862    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/204a59ad-3b92-4bc7-ad14-2dc39aa8a104-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:41.944922 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:41.944872    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/204a59ad-3b92-4bc7-ad14-2dc39aa8a104-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:42.447454 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:42.447424    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425b2345-ctlh2"
Nov 03 08:07:42.447878 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:42.447458    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-425b2345-ctlh2" event={"ID":"204a59ad-3b92-4bc7-ad14-2dc39aa8a104","Type":"ContainerDied","Data":"cf908c437c8063beb42a31f21904426e8069dd25b08693f85acb829a338e745c"}
Nov 03 08:07:42.447878 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:42.447504    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf908c437c8063beb42a31f21904426e8069dd25b08693f85acb829a338e745c"
Nov 03 08:07:42.631620 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:42.631595    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db6d499-75l5p"
Nov 03 08:07:42.755877 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:42.755805    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/588587e7-5d9a-41cf-baa4-1d998c96a47e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/588587e7-5d9a-41cf-baa4-1d998c96a47e-maas-api-xks-ca\") pod \"588587e7-5d9a-41cf-baa4-1d998c96a47e\" (UID: \"588587e7-5d9a-41cf-baa4-1d998c96a47e\") "
Nov 03 08:07:42.756059 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:42.756042    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/588587e7-5d9a-41cf-baa4-1d998c96a47e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/588587e7-5d9a-41cf-baa4-1d998c96a47e-maas-api-openshift-ca\") pod \"588587e7-5d9a-41cf-baa4-1d998c96a47e\" (UID: \"588587e7-5d9a-41cf-baa4-1d998c96a47e\") "
Nov 03 08:07:42.756124 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:42.756095    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/588587e7-5d9a-41cf-baa4-1d998c96a47e-kube-api-access-2hxkn\" (UniqueName: \"kubernetes.io/projected/588587e7-5d9a-41cf-baa4-1d998c96a47e-kube-api-access-2hxkn\") pod \"588587e7-5d9a-41cf-baa4-1d998c96a47e\" (UID: \"588587e7-5d9a-41cf-baa4-1d998c96a47e\") "
Nov 03 08:07:42.756438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:42.756412    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/588587e7-5d9a-41cf-baa4-1d998c96a47e-maas-api-openshift-ca" pod "588587e7-5d9a-41cf-baa4-1d998c96a47e" (UID: "588587e7-5d9a-41cf-baa4-1d998c96a47e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:07:42.756634 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:42.756620    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/588587e7-5d9a-41cf-baa4-1d998c96a47e-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:42.757735 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:42.757706    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/588587e7-5d9a-41cf-baa4-1d998c96a47e-maas-api-xks-ca" pod "588587e7-5d9a-41cf-baa4-1d998c96a47e" (UID: "588587e7-5d9a-41cf-baa4-1d998c96a47e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:07:42.758129 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:42.758100    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/588587e7-5d9a-41cf-baa4-1d998c96a47e-kube-api-access-2hxkn" pod "588587e7-5d9a-41cf-baa4-1d998c96a47e" (UID: "588587e7-5d9a-41cf-baa4-1d998c96a47e"). InnerVolumeSpecName "kube-api-access-2hxkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:07:42.858288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:42.858259    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hxkn\" (UniqueName: \"kubernetes.io/projected/588587e7-5d9a-41cf-baa4-1d998c96a47e-kube-api-access-2hxkn\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:42.858288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:42.858287    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/588587e7-5d9a-41cf-baa4-1d998c96a47e-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:43.318084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.318051    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc"]
Nov 03 08:07:43.318707 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.318693    5070 cpu_manager.go:414] "containerMap: removing container" podUID="3d7b0574-49d7-4b35-94cb-065d525d881a" containerName="revoke-keys"
Nov 03 08:07:43.318771 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.318708    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d7b0574-49d7-4b35-94cb-065d525d881a" containerName="revoke-keys"
Nov 03 08:07:43.318771 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.318747    5070 cpu_manager.go:414] "containerMap: removing container" podUID="204a59ad-3b92-4bc7-ad14-2dc39aa8a104" containerName="revoke-keys"
Nov 03 08:07:43.318771 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.318752    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204a59ad-3b92-4bc7-ad14-2dc39aa8a104" containerName="revoke-keys"
Nov 03 08:07:43.318871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.318782    5070 cpu_manager.go:414] "containerMap: removing container" podUID="3acd647f-bec3-4ace-8674-2c7009f9e91c" containerName="revoke-keys"
Nov 03 08:07:43.318871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.318787    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3acd647f-bec3-4ace-8674-2c7009f9e91c" containerName="revoke-keys"
Nov 03 08:07:43.318871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.318794    5070 cpu_manager.go:414] "containerMap: removing container" podUID="588587e7-5d9a-41cf-baa4-1d998c96a47e" containerName="revoke-keys"
Nov 03 08:07:43.318871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.318799    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="588587e7-5d9a-41cf-baa4-1d998c96a47e" containerName="revoke-keys"
Nov 03 08:07:43.318989 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.318905    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="204a59ad-3b92-4bc7-ad14-2dc39aa8a104" containerName="revoke-keys"
Nov 03 08:07:43.318989 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.318917    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="3acd647f-bec3-4ace-8674-2c7009f9e91c" containerName="revoke-keys"
Nov 03 08:07:43.318989 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.318922    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="588587e7-5d9a-41cf-baa4-1d998c96a47e" containerName="revoke-keys"
Nov 03 08:07:43.318989 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.318929    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d7b0574-49d7-4b35-94cb-065d525d881a" containerName="revoke-keys"
Nov 03 08:07:43.322836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.322811    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc"
Nov 03 08:07:43.329715 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.329690    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc"]
Nov 03 08:07:43.458739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.458711    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db6d499-75l5p"
Nov 03 08:07:43.459217 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.458748    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3db6d499-75l5p" event={"ID":"588587e7-5d9a-41cf-baa4-1d998c96a47e","Type":"ContainerDied","Data":"fe9b8ffa996334a11e40afc821941de53725bd5ab9b316ee6503b229f2059e69"}
Nov 03 08:07:43.459217 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.458829    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe9b8ffa996334a11e40afc821941de53725bd5ab9b316ee6503b229f2059e69"
Nov 03 08:07:43.466607 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.466586    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b91e763-0b36-4d03-bc1e-ccc40d9be701-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d07a8f1f-f86hc\" (UID: \"7b91e763-0b36-4d03-bc1e-ccc40d9be701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc"
Nov 03 08:07:43.466722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.466625    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26dck\" (UniqueName: \"kubernetes.io/projected/7b91e763-0b36-4d03-bc1e-ccc40d9be701-kube-api-access-26dck\") pod \"maas-api-revoke-sub-d07a8f1f-f86hc\" (UID: \"7b91e763-0b36-4d03-bc1e-ccc40d9be701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc"
Nov 03 08:07:43.466872 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.466850    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b91e763-0b36-4d03-bc1e-ccc40d9be701-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d07a8f1f-f86hc\" (UID: \"7b91e763-0b36-4d03-bc1e-ccc40d9be701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc"
Nov 03 08:07:43.568387 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.568326    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b91e763-0b36-4d03-bc1e-ccc40d9be701-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d07a8f1f-f86hc\" (UID: \"7b91e763-0b36-4d03-bc1e-ccc40d9be701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc"
Nov 03 08:07:43.568564 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.568542    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b91e763-0b36-4d03-bc1e-ccc40d9be701-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d07a8f1f-f86hc\" (UID: \"7b91e763-0b36-4d03-bc1e-ccc40d9be701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc"
Nov 03 08:07:43.568645 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.568585    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26dck\" (UniqueName: \"kubernetes.io/projected/7b91e763-0b36-4d03-bc1e-ccc40d9be701-kube-api-access-26dck\") pod \"maas-api-revoke-sub-d07a8f1f-f86hc\" (UID: \"7b91e763-0b36-4d03-bc1e-ccc40d9be701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc"
Nov 03 08:07:43.569071 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.569050    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b91e763-0b36-4d03-bc1e-ccc40d9be701-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d07a8f1f-f86hc\" (UID: \"7b91e763-0b36-4d03-bc1e-ccc40d9be701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc"
Nov 03 08:07:43.570983 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.570961    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b91e763-0b36-4d03-bc1e-ccc40d9be701-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d07a8f1f-f86hc\" (UID: \"7b91e763-0b36-4d03-bc1e-ccc40d9be701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc"
Nov 03 08:07:43.578886 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.578860    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26dck\" (UniqueName: \"kubernetes.io/projected/7b91e763-0b36-4d03-bc1e-ccc40d9be701-kube-api-access-26dck\") pod \"maas-api-revoke-sub-d07a8f1f-f86hc\" (UID: \"7b91e763-0b36-4d03-bc1e-ccc40d9be701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc"
Nov 03 08:07:43.656684 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.656411    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc"
Nov 03 08:07:43.977680 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:07:43.977645    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b91e763_0b36_4d03_bc1e_ccc40d9be701.slice/crio-5accbd835f39690eadf344f822f512b5e179589c3080832ffb289bdb276c0afa WatchSource:0}: Error finding container 5accbd835f39690eadf344f822f512b5e179589c3080832ffb289bdb276c0afa: Status 404 returned error can't find the container with id 5accbd835f39690eadf344f822f512b5e179589c3080832ffb289bdb276c0afa
Nov 03 08:07:43.979924 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:43.979896    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc"]
Nov 03 08:07:44.059608 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:44.059585    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6"]
Nov 03 08:07:44.065407 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:44.065383    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6"
Nov 03 08:07:44.071526 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:44.071502    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6"]
Nov 03 08:07:44.181266 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:44.181231    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6f1097b-a4d2-49fc-9db1-05721fed660d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54bc7452-ztkk6\" (UID: \"b6f1097b-a4d2-49fc-9db1-05721fed660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6"
Nov 03 08:07:44.181411 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:44.181279    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6f1097b-a4d2-49fc-9db1-05721fed660d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54bc7452-ztkk6\" (UID: \"b6f1097b-a4d2-49fc-9db1-05721fed660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6"
Nov 03 08:07:44.181462 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:44.181423    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg4vf\" (UniqueName: \"kubernetes.io/projected/b6f1097b-a4d2-49fc-9db1-05721fed660d-kube-api-access-zg4vf\") pod \"maas-api-revoke-sub-54bc7452-ztkk6\" (UID: \"b6f1097b-a4d2-49fc-9db1-05721fed660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6"
Nov 03 08:07:44.282674 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:44.282647    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg4vf\" (UniqueName: \"kubernetes.io/projected/b6f1097b-a4d2-49fc-9db1-05721fed660d-kube-api-access-zg4vf\") pod \"maas-api-revoke-sub-54bc7452-ztkk6\" (UID: \"b6f1097b-a4d2-49fc-9db1-05721fed660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6"
Nov 03 08:07:44.282887 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:44.282871    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6f1097b-a4d2-49fc-9db1-05721fed660d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54bc7452-ztkk6\" (UID: \"b6f1097b-a4d2-49fc-9db1-05721fed660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6"
Nov 03 08:07:44.282962 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:44.282893    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6f1097b-a4d2-49fc-9db1-05721fed660d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54bc7452-ztkk6\" (UID: \"b6f1097b-a4d2-49fc-9db1-05721fed660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6"
Nov 03 08:07:44.283514 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:44.283489    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6f1097b-a4d2-49fc-9db1-05721fed660d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54bc7452-ztkk6\" (UID: \"b6f1097b-a4d2-49fc-9db1-05721fed660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6"
Nov 03 08:07:44.285060 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:44.285040    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6f1097b-a4d2-49fc-9db1-05721fed660d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54bc7452-ztkk6\" (UID: \"b6f1097b-a4d2-49fc-9db1-05721fed660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6"
Nov 03 08:07:44.288966 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:44.288947    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg4vf\" (UniqueName: \"kubernetes.io/projected/b6f1097b-a4d2-49fc-9db1-05721fed660d-kube-api-access-zg4vf\") pod \"maas-api-revoke-sub-54bc7452-ztkk6\" (UID: \"b6f1097b-a4d2-49fc-9db1-05721fed660d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6"
Nov 03 08:07:44.399997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:44.399970    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6"
Nov 03 08:07:44.470909 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:44.470881    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc" event={"ID":"7b91e763-0b36-4d03-bc1e-ccc40d9be701","Type":"ContainerStarted","Data":"a240c9848b03ebb1243e847f66536317265dfff5bb635e971836523d2dbebd2b"}
Nov 03 08:07:44.471236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:44.470959    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc" event={"ID":"7b91e763-0b36-4d03-bc1e-ccc40d9be701","Type":"ContainerStarted","Data":"5accbd835f39690eadf344f822f512b5e179589c3080832ffb289bdb276c0afa"}
Nov 03 08:07:44.493240 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:44.493188    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc" podStartSLOduration=1.49317383 podStartE2EDuration="1.49317383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:44.485209647 +0000 UTC m=+862.599673582" watchObservedRunningTime="2026-11-03 08:07:44.49317383 +0000 UTC m=+862.607637762"
Nov 03 08:07:44.701609 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:44.701577    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6"]
Nov 03 08:07:45.108450 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:45.108422    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:07:45.481495 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:45.481462    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6" event={"ID":"b6f1097b-a4d2-49fc-9db1-05721fed660d","Type":"ContainerStarted","Data":"db54f21a277799a23b445ad5823b6b73e5525664bbf9b404e0ca28d0b8c5ad89"}
Nov 03 08:07:45.481982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:45.481520    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6" event={"ID":"b6f1097b-a4d2-49fc-9db1-05721fed660d","Type":"ContainerStarted","Data":"fd00a68cdf9fb1c5d43229734d7753044faf3d12f1103c7be1a3633b3fc3b6e9"}
Nov 03 08:07:45.483174 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:45.483144    5070 generic.go:356] "Generic (PLEG): container finished" podID="7b91e763-0b36-4d03-bc1e-ccc40d9be701" containerID="a240c9848b03ebb1243e847f66536317265dfff5bb635e971836523d2dbebd2b" exitCode=0
Nov 03 08:07:45.483265 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:45.483220    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc" event={"ID":"7b91e763-0b36-4d03-bc1e-ccc40d9be701","Type":"ContainerDied","Data":"a240c9848b03ebb1243e847f66536317265dfff5bb635e971836523d2dbebd2b"}
Nov 03 08:07:45.501357 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:45.501310    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6" podStartSLOduration=1.501297299 podStartE2EDuration="1.501297299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:45.496149398 +0000 UTC m=+863.610613329" watchObservedRunningTime="2026-11-03 08:07:45.501297299 +0000 UTC m=+863.615761234"
Nov 03 08:07:46.602817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:46.591369    5070 generic.go:356] "Generic (PLEG): container finished" podID="b6f1097b-a4d2-49fc-9db1-05721fed660d" containerID="db54f21a277799a23b445ad5823b6b73e5525664bbf9b404e0ca28d0b8c5ad89" exitCode=0
Nov 03 08:07:46.602817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:46.592554    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6" event={"ID":"b6f1097b-a4d2-49fc-9db1-05721fed660d","Type":"ContainerDied","Data":"db54f21a277799a23b445ad5823b6b73e5525664bbf9b404e0ca28d0b8c5ad89"}
Nov 03 08:07:47.391939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:47.391890    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:47.735795 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:47.709732    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw"]
Nov 03 08:07:48.482228 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:47.864847    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:48.482228 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:47.864889    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:07:48.482228 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:47.865142    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:07:48.502049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:47.866189    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:49.013549 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:48.225108    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:07:49.015646 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:48.353095    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:49.017109 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:48.419150    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:07:49.022715 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:48.839198    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:07:49.144024 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:49.140882    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:49.236779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:49.235175    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw" podUID="0d26093c-a60a-4cd8-9d43-8ecded62d895" containerName="server" probeResult="failure" output="Get \"https://10.132.0.47:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:49.256592 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:49.256551    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:49.299188 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:49.298850    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9" podUID="917243d5-933a-4f61-9e76-72ec95f6a0dd" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.65:50052\" within 1s: context deadline exceeded"
Nov 03 08:07:49.321818 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:48.905684    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz" podUID="33f324bd-79b1-4540-a090-7e4a9d92f10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:49.392784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:48.972246    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn" podUID="6960b432-0c5c-4c03-bcc5-dec9ff9466c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:49.446783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:49.438859    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m" podUID="cb3722fc-983b-4b33-9269-8e559475aff5" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 03 08:07:49.680114 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:49.481882    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:49.680343 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:49.481967    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:49.700783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:49.681397    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:49.897817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:49.891292    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:07:49.943253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:49.943167    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc"
Nov 03 08:07:49.943472 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:49.943447    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:49.944002 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:49.943954    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt" podUID="f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:49.962535 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:49.962146    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:07:49.962535 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:49.962219    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw"]
Nov 03 08:07:49.962535 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:49.962283    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:07:49.962535 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:49.962324    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:07:49.984964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:49.968874    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw"
Nov 03 08:07:50.035037 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.034994    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:07:50.062170 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.062126    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:07:50.074911 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.071150    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:50.117851 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.085866    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:07:50.117851 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.085932    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:50.178272 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.178223    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:50.295540 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.295444    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd" podUID="cb961482-e401-47f5-8aff-f604b364d631" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.64:50052\" within 1s: context deadline exceeded"
Nov 03 08:07:50.296593 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.296173    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b91e763-0b36-4d03-bc1e-ccc40d9be701-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b91e763-0b36-4d03-bc1e-ccc40d9be701-maas-api-xks-ca\") pod \"7b91e763-0b36-4d03-bc1e-ccc40d9be701\" (UID: \"7b91e763-0b36-4d03-bc1e-ccc40d9be701\") "
Nov 03 08:07:50.297956 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.297930    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b91e763-0b36-4d03-bc1e-ccc40d9be701-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b91e763-0b36-4d03-bc1e-ccc40d9be701-maas-api-openshift-ca\") pod \"7b91e763-0b36-4d03-bc1e-ccc40d9be701\" (UID: \"7b91e763-0b36-4d03-bc1e-ccc40d9be701\") "
Nov 03 08:07:50.298337 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.298318    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b91e763-0b36-4d03-bc1e-ccc40d9be701-kube-api-access-26dck\" (UniqueName: \"kubernetes.io/projected/7b91e763-0b36-4d03-bc1e-ccc40d9be701-kube-api-access-26dck\") pod \"7b91e763-0b36-4d03-bc1e-ccc40d9be701\" (UID: \"7b91e763-0b36-4d03-bc1e-ccc40d9be701\") "
Nov 03 08:07:50.330713 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.330679    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:50.330979 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.330948    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5" podUID="d1a356ec-fba8-45e1-87ba-13950f3e1fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:50.350124 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.350090    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gb5t\" (UniqueName: \"kubernetes.io/projected/3d37485a-b563-4ff7-83e1-decd60b006f7-kube-api-access-4gb5t\") pod \"maas-api-revoke-sub-192e4d3a-k5xdw\" (UID: \"3d37485a-b563-4ff7-83e1-decd60b006f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw"
Nov 03 08:07:50.350574 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.350548    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d37485a-b563-4ff7-83e1-decd60b006f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-192e4d3a-k5xdw\" (UID: \"3d37485a-b563-4ff7-83e1-decd60b006f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw"
Nov 03 08:07:50.350810 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.350791    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d37485a-b563-4ff7-83e1-decd60b006f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-192e4d3a-k5xdw\" (UID: \"3d37485a-b563-4ff7-83e1-decd60b006f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw"
Nov 03 08:07:50.448795 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.448741    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b91e763-0b36-4d03-bc1e-ccc40d9be701-maas-api-openshift-ca" pod "7b91e763-0b36-4d03-bc1e-ccc40d9be701" (UID: "7b91e763-0b36-4d03-bc1e-ccc40d9be701"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:07:50.483785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.477696    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gb5t\" (UniqueName: \"kubernetes.io/projected/3d37485a-b563-4ff7-83e1-decd60b006f7-kube-api-access-4gb5t\") pod \"maas-api-revoke-sub-192e4d3a-k5xdw\" (UID: \"3d37485a-b563-4ff7-83e1-decd60b006f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw"
Nov 03 08:07:50.483785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.477877    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d37485a-b563-4ff7-83e1-decd60b006f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-192e4d3a-k5xdw\" (UID: \"3d37485a-b563-4ff7-83e1-decd60b006f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw"
Nov 03 08:07:50.483785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.477935    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d37485a-b563-4ff7-83e1-decd60b006f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-192e4d3a-k5xdw\" (UID: \"3d37485a-b563-4ff7-83e1-decd60b006f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw"
Nov 03 08:07:50.483785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.478130    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b91e763-0b36-4d03-bc1e-ccc40d9be701-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:50.483785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.479051    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d37485a-b563-4ff7-83e1-decd60b006f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-192e4d3a-k5xdw\" (UID: \"3d37485a-b563-4ff7-83e1-decd60b006f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw"
Nov 03 08:07:50.560184 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:07:50.560097    5070 logging.go:55] [core] [Channel #623 SubChannel #625]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.64:50052", ServerName: "10.132.0.64:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.64:50052: operation was canceled"
Nov 03 08:07:50.708276 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.708237    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:50.708521 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.708490    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn" podUID="4fc257d6-34f4-412a-80c4-9a7d4462d2f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:50.856618 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.856558    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b91e763-0b36-4d03-bc1e-ccc40d9be701-kube-api-access-26dck" pod "7b91e763-0b36-4d03-bc1e-ccc40d9be701" (UID: "7b91e763-0b36-4d03-bc1e-ccc40d9be701"). InnerVolumeSpecName "kube-api-access-26dck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:07:50.858377 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.858326    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b91e763-0b36-4d03-bc1e-ccc40d9be701-maas-api-xks-ca" pod "7b91e763-0b36-4d03-bc1e-ccc40d9be701" (UID: "7b91e763-0b36-4d03-bc1e-ccc40d9be701"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:07:50.927942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.926502    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gb5t\" (UniqueName: \"kubernetes.io/projected/3d37485a-b563-4ff7-83e1-decd60b006f7-kube-api-access-4gb5t\") pod \"maas-api-revoke-sub-192e4d3a-k5xdw\" (UID: \"3d37485a-b563-4ff7-83e1-decd60b006f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw"
Nov 03 08:07:50.958783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.928687    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded"
Nov 03 08:07:50.958783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.928942    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:07:50.958783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.929554    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:50.958783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.938913    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:07:50.958783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.940942    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b91e763-0b36-4d03-bc1e-ccc40d9be701-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:50.958783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.940967    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26dck\" (UniqueName: \"kubernetes.io/projected/7b91e763-0b36-4d03-bc1e-ccc40d9be701-kube-api-access-26dck\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:50.974010 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:07:50.972642    5070 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.01s"
Nov 03 08:07:50.974010 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.972718    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:07:50.974010 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:50.972794    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:07:51.112876 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:51.073624    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:51.125785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:51.114153    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:07:51.163783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:51.127644    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:07:51.203783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:51.177945    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:07:51.228784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:51.204368    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:07:51.243782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:51.166629    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d37485a-b563-4ff7-83e1-decd60b006f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-192e4d3a-k5xdw\" (UID: \"3d37485a-b563-4ff7-83e1-decd60b006f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw"
Nov 03 08:07:51.350790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:51.330887    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr"]
Nov 03 08:07:51.473955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:51.331272    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:51.473955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:51.468222    5070 cpu_manager.go:414] "containerMap: removing container" podUID="7b91e763-0b36-4d03-bc1e-ccc40d9be701" containerName="revoke-keys"
Nov 03 08:07:51.473955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:51.468242    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b91e763-0b36-4d03-bc1e-ccc40d9be701" containerName="revoke-keys"
Nov 03 08:07:51.473955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:51.468296    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw"
Nov 03 08:07:51.473955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:51.468504    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b91e763-0b36-4d03-bc1e-ccc40d9be701" containerName="revoke-keys"
Nov 03 08:07:51.474935 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:51.474910    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc"
Nov 03 08:07:51.932933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:51.932887    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:51.976782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:51.947926    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:07:51.976782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:51.948816    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:07:52.031840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:52.004017    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:07:52.031840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:52.004084    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:52.148781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:52.147888    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:52.209972 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:52.182339    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:52.370832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:52.363669    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:52.508712 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:52.416326    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:07:52.508712 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:52.438961    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:07:52.508712 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:52.470480    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:07:52.633378 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:52.633283    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:07:52.797928 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:52.775459    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb98918b-b331-464e-88d2-081c8dcb7807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e46eb63-ff7pr\" (UID: \"bb98918b-b331-464e-88d2-081c8dcb7807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr"
Nov 03 08:07:52.797928 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:52.775547    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb98918b-b331-464e-88d2-081c8dcb7807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e46eb63-ff7pr\" (UID: \"bb98918b-b331-464e-88d2-081c8dcb7807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr"
Nov 03 08:07:52.797928 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:52.775690    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgc7v\" (UniqueName: \"kubernetes.io/projected/bb98918b-b331-464e-88d2-081c8dcb7807-kube-api-access-hgc7v\") pod \"maas-api-revoke-sub-6e46eb63-ff7pr\" (UID: \"bb98918b-b331-464e-88d2-081c8dcb7807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr"
Nov 03 08:07:52.943834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:52.939844    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:52.968953 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:52.968910    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:52.975785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:52.969557    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:07:52.992779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:52.976694    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb98918b-b331-464e-88d2-081c8dcb7807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e46eb63-ff7pr\" (UID: \"bb98918b-b331-464e-88d2-081c8dcb7807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr"
Nov 03 08:07:52.993084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:52.993056    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb98918b-b331-464e-88d2-081c8dcb7807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e46eb63-ff7pr\" (UID: \"bb98918b-b331-464e-88d2-081c8dcb7807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr"
Nov 03 08:07:52.993382 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:52.993361    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgc7v\" (UniqueName: \"kubernetes.io/projected/bb98918b-b331-464e-88d2-081c8dcb7807-kube-api-access-hgc7v\") pod \"maas-api-revoke-sub-6e46eb63-ff7pr\" (UID: \"bb98918b-b331-464e-88d2-081c8dcb7807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr"
Nov 03 08:07:53.027779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:53.019835    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb98918b-b331-464e-88d2-081c8dcb7807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e46eb63-ff7pr\" (UID: \"bb98918b-b331-464e-88d2-081c8dcb7807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr"
Nov 03 08:07:53.077270 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:53.043355    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb98918b-b331-464e-88d2-081c8dcb7807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e46eb63-ff7pr\" (UID: \"bb98918b-b331-464e-88d2-081c8dcb7807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr"
Nov 03 08:07:53.092624 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:53.092585    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgc7v\" (UniqueName: \"kubernetes.io/projected/bb98918b-b331-464e-88d2-081c8dcb7807-kube-api-access-hgc7v\") pod \"maas-api-revoke-sub-6e46eb63-ff7pr\" (UID: \"bb98918b-b331-464e-88d2-081c8dcb7807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr"
Nov 03 08:07:53.288077 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:07:53.287973    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b91e763_0b36_4d03_bc1e_ccc40d9be701.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 08:07:53.403780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:53.396713    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:07:53.602164 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:53.473929    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:07:53.634087 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:53.474096    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:07:53.634087 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:53.474949    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:53.634087 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:53.495628    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc" event={"ID":"7b91e763-0b36-4d03-bc1e-ccc40d9be701","Type":"ContainerDied","Data":"5accbd835f39690eadf344f822f512b5e179589c3080832ffb289bdb276c0afa"}
Nov 03 08:07:53.634087 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:53.613802    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5accbd835f39690eadf344f822f512b5e179589c3080832ffb289bdb276c0afa"
Nov 03 08:07:53.634087 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:53.614022    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:07:53.634087 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:53.614050    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr"]
Nov 03 08:07:53.634087 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:53.528275    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr"
Nov 03 08:07:53.634087 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:53.536724    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:07:53.883144 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:07:53.882950    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b91e763_0b36_4d03_bc1e_ccc40d9be701.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 08:07:54.022870 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:07:54.022834    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:07:54.069617 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.067874    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6" event={"ID":"b6f1097b-a4d2-49fc-9db1-05721fed660d","Type":"ContainerDied","Data":"fd00a68cdf9fb1c5d43229734d7753044faf3d12f1103c7be1a3633b3fc3b6e9"}
Nov 03 08:07:54.069617 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.067940    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd00a68cdf9fb1c5d43229734d7753044faf3d12f1103c7be1a3633b3fc3b6e9"
Nov 03 08:07:54.069617 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.068061    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:07:54.118063 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.118037    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:07:54.118245 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.118229    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:07:54.118386 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.118373    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:07:54.118521 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.118508    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:07:54.118646 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.118634    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:07:54.118790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.118749    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:07:54.118935 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.118922    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:07:54.119059 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.119047    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:07:54.121744 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.120933    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw"]
Nov 03 08:07:54.126478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.125805    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6"
Nov 03 08:07:54.134296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.133805    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr"
Nov 03 08:07:54.134296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.133951    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:07:54.135083 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.135056    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:07:54.139245 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.139221    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:07:54.203241 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.203211    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6f1097b-a4d2-49fc-9db1-05721fed660d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6f1097b-a4d2-49fc-9db1-05721fed660d-maas-api-xks-ca\") pod \"b6f1097b-a4d2-49fc-9db1-05721fed660d\" (UID: \"b6f1097b-a4d2-49fc-9db1-05721fed660d\") "
Nov 03 08:07:54.203440 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.203424    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6f1097b-a4d2-49fc-9db1-05721fed660d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6f1097b-a4d2-49fc-9db1-05721fed660d-maas-api-openshift-ca\") pod \"b6f1097b-a4d2-49fc-9db1-05721fed660d\" (UID: \"b6f1097b-a4d2-49fc-9db1-05721fed660d\") "
Nov 03 08:07:54.203509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.203473    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6f1097b-a4d2-49fc-9db1-05721fed660d-kube-api-access-zg4vf\" (UniqueName: \"kubernetes.io/projected/b6f1097b-a4d2-49fc-9db1-05721fed660d-kube-api-access-zg4vf\") pod \"b6f1097b-a4d2-49fc-9db1-05721fed660d\" (UID: \"b6f1097b-a4d2-49fc-9db1-05721fed660d\") "
Nov 03 08:07:54.214894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.210013    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f1097b-a4d2-49fc-9db1-05721fed660d-kube-api-access-zg4vf" pod "b6f1097b-a4d2-49fc-9db1-05721fed660d" (UID: "b6f1097b-a4d2-49fc-9db1-05721fed660d"). InnerVolumeSpecName "kube-api-access-zg4vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:07:54.214894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.211123    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6f1097b-a4d2-49fc-9db1-05721fed660d-maas-api-xks-ca" pod "b6f1097b-a4d2-49fc-9db1-05721fed660d" (UID: "b6f1097b-a4d2-49fc-9db1-05721fed660d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:07:54.214894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.211281    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f1097b-a4d2-49fc-9db1-05721fed660d-maas-api-openshift-ca" pod "b6f1097b-a4d2-49fc-9db1-05721fed660d" (UID: "b6f1097b-a4d2-49fc-9db1-05721fed660d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:07:54.306932 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.305319    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6f1097b-a4d2-49fc-9db1-05721fed660d-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:54.306932 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.305347    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg4vf\" (UniqueName: \"kubernetes.io/projected/b6f1097b-a4d2-49fc-9db1-05721fed660d-kube-api-access-zg4vf\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:54.306932 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.305357    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6f1097b-a4d2-49fc-9db1-05721fed660d-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:54.492890 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:54.492745    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr"]
Nov 03 08:07:54.502359 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:07:54.502330    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb98918b_b331_464e_88d2_081c8dcb7807.slice/crio-96bc2881a12481d664cbb4dbe2d41aa72828acd103b60941a06fd15cb2d40aeb WatchSource:0}: Error finding container 96bc2881a12481d664cbb4dbe2d41aa72828acd103b60941a06fd15cb2d40aeb: Status 404 returned error can't find the container with id 96bc2881a12481d664cbb4dbe2d41aa72828acd103b60941a06fd15cb2d40aeb
Nov 03 08:07:55.048271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:55.048234    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr" event={"ID":"bb98918b-b331-464e-88d2-081c8dcb7807","Type":"ContainerStarted","Data":"00ac02669fdb0661440f0185da90b3a51d6cd62fe74be834b83dbfb161fc09ba"}
Nov 03 08:07:55.048418 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:55.048297    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr" event={"ID":"bb98918b-b331-464e-88d2-081c8dcb7807","Type":"ContainerStarted","Data":"96bc2881a12481d664cbb4dbe2d41aa72828acd103b60941a06fd15cb2d40aeb"}
Nov 03 08:07:55.050218 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:55.050186    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw" event={"ID":"3d37485a-b563-4ff7-83e1-decd60b006f7","Type":"ContainerStarted","Data":"82a2dca6a9eca705a350bb91693d8488079f4502485f617b33466a0b6902ae42"}
Nov 03 08:07:55.050348 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:55.050238    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw" event={"ID":"3d37485a-b563-4ff7-83e1-decd60b006f7","Type":"ContainerStarted","Data":"2bcf7f294d77ef6f36b193760fc577c435f5a5e1ff1e76db885bc69a64d94a2d"}
Nov 03 08:07:55.050966 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:55.050944    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6"
Nov 03 08:07:55.066381 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:55.066324    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr" podStartSLOduration=6.06630731 podStartE2EDuration="6.06630731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:55.060600237 +0000 UTC m=+873.175064182" watchObservedRunningTime="2026-11-03 08:07:55.06630731 +0000 UTC m=+873.180771242"
Nov 03 08:07:55.086922 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:55.086874    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw" podStartSLOduration=8.086857995 podStartE2EDuration="8.086857995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:07:55.078003335 +0000 UTC m=+873.192467266" watchObservedRunningTime="2026-11-03 08:07:55.086857995 +0000 UTC m=+873.201321926"
Nov 03 08:07:56.060860 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:56.060821    5070 generic.go:356] "Generic (PLEG): container finished" podID="bb98918b-b331-464e-88d2-081c8dcb7807" containerID="00ac02669fdb0661440f0185da90b3a51d6cd62fe74be834b83dbfb161fc09ba" exitCode=0
Nov 03 08:07:56.061302 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:56.060899    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr" event={"ID":"bb98918b-b331-464e-88d2-081c8dcb7807","Type":"ContainerDied","Data":"00ac02669fdb0661440f0185da90b3a51d6cd62fe74be834b83dbfb161fc09ba"}
Nov 03 08:07:56.062808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:56.062748    5070 generic.go:356] "Generic (PLEG): container finished" podID="3d37485a-b563-4ff7-83e1-decd60b006f7" containerID="82a2dca6a9eca705a350bb91693d8488079f4502485f617b33466a0b6902ae42" exitCode=0
Nov 03 08:07:56.063140 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:56.062826    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw" event={"ID":"3d37485a-b563-4ff7-83e1-decd60b006f7","Type":"ContainerDied","Data":"82a2dca6a9eca705a350bb91693d8488079f4502485f617b33466a0b6902ae42"}
Nov 03 08:07:57.286228 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:57.286196    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr"
Nov 03 08:07:57.291618 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:57.291594    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw"
Nov 03 08:07:57.373159 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:57.373131    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb98918b-b331-464e-88d2-081c8dcb7807-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb98918b-b331-464e-88d2-081c8dcb7807-maas-api-xks-ca\") pod \"bb98918b-b331-464e-88d2-081c8dcb7807\" (UID: \"bb98918b-b331-464e-88d2-081c8dcb7807\") "
Nov 03 08:07:57.373299 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:57.373207    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d37485a-b563-4ff7-83e1-decd60b006f7-kube-api-access-4gb5t\" (UniqueName: \"kubernetes.io/projected/3d37485a-b563-4ff7-83e1-decd60b006f7-kube-api-access-4gb5t\") pod \"3d37485a-b563-4ff7-83e1-decd60b006f7\" (UID: \"3d37485a-b563-4ff7-83e1-decd60b006f7\") "
Nov 03 08:07:57.373299 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:57.373277    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d37485a-b563-4ff7-83e1-decd60b006f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d37485a-b563-4ff7-83e1-decd60b006f7-maas-api-openshift-ca\") pod \"3d37485a-b563-4ff7-83e1-decd60b006f7\" (UID: \"3d37485a-b563-4ff7-83e1-decd60b006f7\") "
Nov 03 08:07:57.373434 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:57.373413    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb98918b-b331-464e-88d2-081c8dcb7807-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb98918b-b331-464e-88d2-081c8dcb7807-maas-api-openshift-ca\") pod \"bb98918b-b331-464e-88d2-081c8dcb7807\" (UID: \"bb98918b-b331-464e-88d2-081c8dcb7807\") "
Nov 03 08:07:57.373526 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:57.373507    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d37485a-b563-4ff7-83e1-decd60b006f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d37485a-b563-4ff7-83e1-decd60b006f7-maas-api-xks-ca\") pod \"3d37485a-b563-4ff7-83e1-decd60b006f7\" (UID: \"3d37485a-b563-4ff7-83e1-decd60b006f7\") "
Nov 03 08:07:57.373648 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:57.373590    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d37485a-b563-4ff7-83e1-decd60b006f7-maas-api-openshift-ca" pod "3d37485a-b563-4ff7-83e1-decd60b006f7" (UID: "3d37485a-b563-4ff7-83e1-decd60b006f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:07:57.373804 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:57.373738    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb98918b-b331-464e-88d2-081c8dcb7807-maas-api-openshift-ca" pod "bb98918b-b331-464e-88d2-081c8dcb7807" (UID: "bb98918b-b331-464e-88d2-081c8dcb7807"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:07:57.373913 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:57.373754    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb98918b-b331-464e-88d2-081c8dcb7807-kube-api-access-hgc7v\" (UniqueName: \"kubernetes.io/projected/bb98918b-b331-464e-88d2-081c8dcb7807-kube-api-access-hgc7v\") pod \"bb98918b-b331-464e-88d2-081c8dcb7807\" (UID: \"bb98918b-b331-464e-88d2-081c8dcb7807\") "
Nov 03 08:07:57.374531 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:57.374503    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d37485a-b563-4ff7-83e1-decd60b006f7-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:57.374649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:57.374534    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb98918b-b331-464e-88d2-081c8dcb7807-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:57.375735 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:57.375712    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb98918b-b331-464e-88d2-081c8dcb7807-maas-api-xks-ca" pod "bb98918b-b331-464e-88d2-081c8dcb7807" (UID: "bb98918b-b331-464e-88d2-081c8dcb7807"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:07:57.376296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:57.376263    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb98918b-b331-464e-88d2-081c8dcb7807-kube-api-access-hgc7v" pod "bb98918b-b331-464e-88d2-081c8dcb7807" (UID: "bb98918b-b331-464e-88d2-081c8dcb7807"). InnerVolumeSpecName "kube-api-access-hgc7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:07:57.376483 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:57.376458    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d37485a-b563-4ff7-83e1-decd60b006f7-maas-api-xks-ca" pod "3d37485a-b563-4ff7-83e1-decd60b006f7" (UID: "3d37485a-b563-4ff7-83e1-decd60b006f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:07:57.376617 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:57.376597    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d37485a-b563-4ff7-83e1-decd60b006f7-kube-api-access-4gb5t" pod "3d37485a-b563-4ff7-83e1-decd60b006f7" (UID: "3d37485a-b563-4ff7-83e1-decd60b006f7"). InnerVolumeSpecName "kube-api-access-4gb5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:07:57.476420 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:57.476390    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gb5t\" (UniqueName: \"kubernetes.io/projected/3d37485a-b563-4ff7-83e1-decd60b006f7-kube-api-access-4gb5t\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:57.476420 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:57.476420    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d37485a-b563-4ff7-83e1-decd60b006f7-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:57.476579 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:57.476430    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgc7v\" (UniqueName: \"kubernetes.io/projected/bb98918b-b331-464e-88d2-081c8dcb7807-kube-api-access-hgc7v\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:57.476579 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:57.476439    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb98918b-b331-464e-88d2-081c8dcb7807-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:07:58.090600 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.090450    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr" event={"ID":"bb98918b-b331-464e-88d2-081c8dcb7807","Type":"ContainerDied","Data":"96bc2881a12481d664cbb4dbe2d41aa72828acd103b60941a06fd15cb2d40aeb"}
Nov 03 08:07:58.090600 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.090514    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96bc2881a12481d664cbb4dbe2d41aa72828acd103b60941a06fd15cb2d40aeb"
Nov 03 08:07:58.090600 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.090599    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr"
Nov 03 08:07:58.094194 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.094161    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw" event={"ID":"3d37485a-b563-4ff7-83e1-decd60b006f7","Type":"ContainerDied","Data":"2bcf7f294d77ef6f36b193760fc577c435f5a5e1ff1e76db885bc69a64d94a2d"}
Nov 03 08:07:58.094406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.094214    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bcf7f294d77ef6f36b193760fc577c435f5a5e1ff1e76db885bc69a64d94a2d"
Nov 03 08:07:58.094406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.094270    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw"
Nov 03 08:07:58.518376 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.518342    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g"]
Nov 03 08:07:58.519123 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.519105    5070 cpu_manager.go:414] "containerMap: removing container" podUID="b6f1097b-a4d2-49fc-9db1-05721fed660d" containerName="revoke-keys"
Nov 03 08:07:58.519190 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.519124    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f1097b-a4d2-49fc-9db1-05721fed660d" containerName="revoke-keys"
Nov 03 08:07:58.519190 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.519141    5070 cpu_manager.go:414] "containerMap: removing container" podUID="bb98918b-b331-464e-88d2-081c8dcb7807" containerName="revoke-keys"
Nov 03 08:07:58.519190 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.519146    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb98918b-b331-464e-88d2-081c8dcb7807" containerName="revoke-keys"
Nov 03 08:07:58.519190 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.519160    5070 cpu_manager.go:414] "containerMap: removing container" podUID="3d37485a-b563-4ff7-83e1-decd60b006f7" containerName="revoke-keys"
Nov 03 08:07:58.519190 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.519166    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d37485a-b563-4ff7-83e1-decd60b006f7" containerName="revoke-keys"
Nov 03 08:07:58.519349 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.519278    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6f1097b-a4d2-49fc-9db1-05721fed660d" containerName="revoke-keys"
Nov 03 08:07:58.519349 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.519286    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb98918b-b331-464e-88d2-081c8dcb7807" containerName="revoke-keys"
Nov 03 08:07:58.519349 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.519293    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d37485a-b563-4ff7-83e1-decd60b006f7" containerName="revoke-keys"
Nov 03 08:07:58.523482 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.523459    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g"
Nov 03 08:07:58.526451 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.526192    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:07:58.526451 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.526335    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:07:58.526451 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.526399    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:07:58.526451 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.526423    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:07:58.530487 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.530464    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g"]
Nov 03 08:07:58.593400 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.593374    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcr85\" (UniqueName: \"kubernetes.io/projected/e1653421-2455-40e3-9103-f512bb881f04-kube-api-access-jcr85\") pod \"maas-api-revoke-sub-bf71ce56-zn65g\" (UID: \"e1653421-2455-40e3-9103-f512bb881f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g"
Nov 03 08:07:58.593526 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.593511    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1653421-2455-40e3-9103-f512bb881f04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf71ce56-zn65g\" (UID: \"e1653421-2455-40e3-9103-f512bb881f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g"
Nov 03 08:07:58.593576 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.593564    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1653421-2455-40e3-9103-f512bb881f04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf71ce56-zn65g\" (UID: \"e1653421-2455-40e3-9103-f512bb881f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g"
Nov 03 08:07:58.695023 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.694998    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcr85\" (UniqueName: \"kubernetes.io/projected/e1653421-2455-40e3-9103-f512bb881f04-kube-api-access-jcr85\") pod \"maas-api-revoke-sub-bf71ce56-zn65g\" (UID: \"e1653421-2455-40e3-9103-f512bb881f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g"
Nov 03 08:07:58.695148 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.695098    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1653421-2455-40e3-9103-f512bb881f04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf71ce56-zn65g\" (UID: \"e1653421-2455-40e3-9103-f512bb881f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g"
Nov 03 08:07:58.695192 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.695156    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1653421-2455-40e3-9103-f512bb881f04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf71ce56-zn65g\" (UID: \"e1653421-2455-40e3-9103-f512bb881f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g"
Nov 03 08:07:58.695860 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.695840    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1653421-2455-40e3-9103-f512bb881f04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf71ce56-zn65g\" (UID: \"e1653421-2455-40e3-9103-f512bb881f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g"
Nov 03 08:07:58.697457 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.697435    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1653421-2455-40e3-9103-f512bb881f04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf71ce56-zn65g\" (UID: \"e1653421-2455-40e3-9103-f512bb881f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g"
Nov 03 08:07:58.704735 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.704709    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcr85\" (UniqueName: \"kubernetes.io/projected/e1653421-2455-40e3-9103-f512bb881f04-kube-api-access-jcr85\") pod \"maas-api-revoke-sub-bf71ce56-zn65g\" (UID: \"e1653421-2455-40e3-9103-f512bb881f04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g"
Nov 03 08:07:58.840322 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:58.840251    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g"
Nov 03 08:07:59.152896 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:07:59.149299    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g"]
Nov 03 08:07:59.154347 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:07:59.154316    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1653421_2455_40e3_9103_f512bb881f04.slice/crio-86ca5acf956d704abfe48e0ec8a2cd6164f49f317a1adc740a44621d0d804bcc WatchSource:0}: Error finding container 86ca5acf956d704abfe48e0ec8a2cd6164f49f317a1adc740a44621d0d804bcc: Status 404 returned error can't find the container with id 86ca5acf956d704abfe48e0ec8a2cd6164f49f317a1adc740a44621d0d804bcc
Nov 03 08:08:00.122938 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:00.122888    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g" event={"ID":"e1653421-2455-40e3-9103-f512bb881f04","Type":"ContainerStarted","Data":"9cb8e860a4d04f080559fdf7246c2c01849e61370c90cfc4d54e019669734ce0"}
Nov 03 08:08:00.123313 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:00.122968    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g" event={"ID":"e1653421-2455-40e3-9103-f512bb881f04","Type":"ContainerStarted","Data":"86ca5acf956d704abfe48e0ec8a2cd6164f49f317a1adc740a44621d0d804bcc"}
Nov 03 08:08:00.139679 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:00.139632    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g" podStartSLOduration=2.13962009 podStartE2EDuration="2.13962009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:00.135730404 +0000 UTC m=+878.250194335" watchObservedRunningTime="2026-11-03 08:08:00.13962009 +0000 UTC m=+878.254084021"
Nov 03 08:08:01.134413 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.134371    5070 generic.go:356] "Generic (PLEG): container finished" podID="e1653421-2455-40e3-9103-f512bb881f04" containerID="9cb8e860a4d04f080559fdf7246c2c01849e61370c90cfc4d54e019669734ce0" exitCode=0
Nov 03 08:08:01.134894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.134473    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g" event={"ID":"e1653421-2455-40e3-9103-f512bb881f04","Type":"ContainerDied","Data":"9cb8e860a4d04f080559fdf7246c2c01849e61370c90cfc4d54e019669734ce0"}
Nov 03 08:08:01.229305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.229260    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"]
Nov 03 08:08:01.236702 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.236674    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.237784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.237747    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"]
Nov 03 08:08:01.338338 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.338302    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.338459 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.338346    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0c52e6-b384-411f-9c3a-c2d76c149238-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.338496 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.338477    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.338552 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.338530    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz2vf\" (UniqueName: \"kubernetes.io/projected/7f0c52e6-b384-411f-9c3a-c2d76c149238-kube-api-access-jz2vf\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.338671 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.338654    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.338719 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.338708    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.338769 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.338727    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.338769 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.338752    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.338838 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.338799    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.439705 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.439640    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.439845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.439713    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.439845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.439737    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0c52e6-b384-411f-9c3a-c2d76c149238-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.439938 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.439862    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.439938 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.439923    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz2vf\" (UniqueName: \"kubernetes.io/projected/7f0c52e6-b384-411f-9c3a-c2d76c149238-kube-api-access-jz2vf\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.440061 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.440042    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.440130 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.440117    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.440179 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.440141    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.440239 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.440170    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.440841 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.440452    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-workload-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.440841 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.440579    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-data\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.440841 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.440625    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0c52e6-b384-411f-9c3a-c2d76c149238-istiod-ca-cert\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.440841 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.440703    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-workload-certs\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.440841 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.440795    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-credential-socket\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.442271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.442247    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-envoy\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.442485 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.442469    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-podinfo\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.446441 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.446417    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-token\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.446542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.446520    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz2vf\" (UniqueName: \"kubernetes.io/projected/7f0c52e6-b384-411f-9c3a-c2d76c149238-kube-api-access-jz2vf\") pod \"e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") " pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.552223 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.552182    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:01.760630 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.760561    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2"]
Nov 03 08:08:01.766636 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.766609    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2"
Nov 03 08:08:01.778880 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.777570    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2"]
Nov 03 08:08:01.847990 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.847959    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a67dfc8-06fa-4be2-9be2-40477be1c36c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdb5bb3f-bn2f2\" (UID: \"3a67dfc8-06fa-4be2-9be2-40477be1c36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2"
Nov 03 08:08:01.848161 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.848027    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a67dfc8-06fa-4be2-9be2-40477be1c36c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdb5bb3f-bn2f2\" (UID: \"3a67dfc8-06fa-4be2-9be2-40477be1c36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2"
Nov 03 08:08:01.848161 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.848053    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-strnj\" (UniqueName: \"kubernetes.io/projected/3a67dfc8-06fa-4be2-9be2-40477be1c36c-kube-api-access-strnj\") pod \"maas-api-revoke-sub-fdb5bb3f-bn2f2\" (UID: \"3a67dfc8-06fa-4be2-9be2-40477be1c36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2"
Nov 03 08:08:01.873361 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.873327    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"]
Nov 03 08:08:01.878485 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:08:01.878454    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0c52e6_b384_411f_9c3a_c2d76c149238.slice/crio-f03129ecc759485af14ce97800b02658cab12775680741e7d372ff3eda0e31dd WatchSource:0}: Error finding container f03129ecc759485af14ce97800b02658cab12775680741e7d372ff3eda0e31dd: Status 404 returned error can't find the container with id f03129ecc759485af14ce97800b02658cab12775680741e7d372ff3eda0e31dd
Nov 03 08:08:01.894947 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.894900    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:08:01.895214 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.895188    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:08:01.895430 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.895405    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:08:01.949052 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.949025    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a67dfc8-06fa-4be2-9be2-40477be1c36c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdb5bb3f-bn2f2\" (UID: \"3a67dfc8-06fa-4be2-9be2-40477be1c36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2"
Nov 03 08:08:01.949178 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.949120    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a67dfc8-06fa-4be2-9be2-40477be1c36c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdb5bb3f-bn2f2\" (UID: \"3a67dfc8-06fa-4be2-9be2-40477be1c36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2"
Nov 03 08:08:01.949178 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.949172    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-strnj\" (UniqueName: \"kubernetes.io/projected/3a67dfc8-06fa-4be2-9be2-40477be1c36c-kube-api-access-strnj\") pod \"maas-api-revoke-sub-fdb5bb3f-bn2f2\" (UID: \"3a67dfc8-06fa-4be2-9be2-40477be1c36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2"
Nov 03 08:08:01.949674 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.949648    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a67dfc8-06fa-4be2-9be2-40477be1c36c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdb5bb3f-bn2f2\" (UID: \"3a67dfc8-06fa-4be2-9be2-40477be1c36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2"
Nov 03 08:08:01.951640 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.951606    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a67dfc8-06fa-4be2-9be2-40477be1c36c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdb5bb3f-bn2f2\" (UID: \"3a67dfc8-06fa-4be2-9be2-40477be1c36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2"
Nov 03 08:08:01.957196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:01.956946    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-strnj\" (UniqueName: \"kubernetes.io/projected/3a67dfc8-06fa-4be2-9be2-40477be1c36c-kube-api-access-strnj\") pod \"maas-api-revoke-sub-fdb5bb3f-bn2f2\" (UID: \"3a67dfc8-06fa-4be2-9be2-40477be1c36c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2"
Nov 03 08:08:02.146925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:02.146896    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2"
Nov 03 08:08:02.148010 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:02.147936    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6" event={"ID":"7f0c52e6-b384-411f-9c3a-c2d76c149238","Type":"ContainerStarted","Data":"e719b3e626dc6538cd9eca46d8bad4fa011b80318345a2b254b98549ea65baf1"}
Nov 03 08:08:02.148010 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:02.147992    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6" event={"ID":"7f0c52e6-b384-411f-9c3a-c2d76c149238","Type":"ContainerStarted","Data":"f03129ecc759485af14ce97800b02658cab12775680741e7d372ff3eda0e31dd"}
Nov 03 08:08:02.187128 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:02.187071    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6" podStartSLOduration=1.187052136 podStartE2EDuration="1.187052136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:02.173072366 +0000 UTC m=+880.287536327" watchObservedRunningTime="2026-11-03 08:08:02.187052136 +0000 UTC m=+880.301516068"
Nov 03 08:08:02.343033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:02.343005    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g"
Nov 03 08:08:02.450574 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:08:02.450526    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a67dfc8_06fa_4be2_9be2_40477be1c36c.slice/crio-4104cee45e0196235a0259f2d788348741df0f7baf167814a02bf41ae30fa6aa WatchSource:0}: Error finding container 4104cee45e0196235a0259f2d788348741df0f7baf167814a02bf41ae30fa6aa: Status 404 returned error can't find the container with id 4104cee45e0196235a0259f2d788348741df0f7baf167814a02bf41ae30fa6aa
Nov 03 08:08:02.452210 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:02.452185    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2"]
Nov 03 08:08:02.456418 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:02.456400    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1653421-2455-40e3-9103-f512bb881f04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1653421-2455-40e3-9103-f512bb881f04-maas-api-openshift-ca\") pod \"e1653421-2455-40e3-9103-f512bb881f04\" (UID: \"e1653421-2455-40e3-9103-f512bb881f04\") "
Nov 03 08:08:02.456507 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:02.456455    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1653421-2455-40e3-9103-f512bb881f04-kube-api-access-jcr85\" (UniqueName: \"kubernetes.io/projected/e1653421-2455-40e3-9103-f512bb881f04-kube-api-access-jcr85\") pod \"e1653421-2455-40e3-9103-f512bb881f04\" (UID: \"e1653421-2455-40e3-9103-f512bb881f04\") "
Nov 03 08:08:02.456507 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:02.456488    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1653421-2455-40e3-9103-f512bb881f04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1653421-2455-40e3-9103-f512bb881f04-maas-api-xks-ca\") pod \"e1653421-2455-40e3-9103-f512bb881f04\" (UID: \"e1653421-2455-40e3-9103-f512bb881f04\") "
Nov 03 08:08:02.456813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:02.456750    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1653421-2455-40e3-9103-f512bb881f04-maas-api-openshift-ca" pod "e1653421-2455-40e3-9103-f512bb881f04" (UID: "e1653421-2455-40e3-9103-f512bb881f04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:08:02.457231 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:02.457206    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1653421-2455-40e3-9103-f512bb881f04-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:08:02.458742 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:02.458720    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1653421-2455-40e3-9103-f512bb881f04-kube-api-access-jcr85" pod "e1653421-2455-40e3-9103-f512bb881f04" (UID: "e1653421-2455-40e3-9103-f512bb881f04"). InnerVolumeSpecName "kube-api-access-jcr85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:08:02.459112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:02.459091    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1653421-2455-40e3-9103-f512bb881f04-maas-api-xks-ca" pod "e1653421-2455-40e3-9103-f512bb881f04" (UID: "e1653421-2455-40e3-9103-f512bb881f04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:08:02.553044 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:02.553008    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:02.554585 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:02.554553    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 08:08:02.554698 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:02.554609    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6" podUID="7f0c52e6-b384-411f-9c3a-c2d76c149238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 08:08:02.559198 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:02.559167    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcr85\" (UniqueName: \"kubernetes.io/projected/e1653421-2455-40e3-9103-f512bb881f04-kube-api-access-jcr85\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:08:02.559300 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:02.559202    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1653421-2455-40e3-9103-f512bb881f04-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:08:03.174832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:03.167439    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g" event={"ID":"e1653421-2455-40e3-9103-f512bb881f04","Type":"ContainerDied","Data":"86ca5acf956d704abfe48e0ec8a2cd6164f49f317a1adc740a44621d0d804bcc"}
Nov 03 08:08:03.174832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:03.167516    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86ca5acf956d704abfe48e0ec8a2cd6164f49f317a1adc740a44621d0d804bcc"
Nov 03 08:08:03.174832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:03.167651    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g"
Nov 03 08:08:03.174832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:03.173684    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2" event={"ID":"3a67dfc8-06fa-4be2-9be2-40477be1c36c","Type":"ContainerStarted","Data":"97f795f398132977c14fd2f64b5399a3a3a2f2d7730c616628529acc52ae721f"}
Nov 03 08:08:03.174832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:03.173737    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2" event={"ID":"3a67dfc8-06fa-4be2-9be2-40477be1c36c","Type":"ContainerStarted","Data":"4104cee45e0196235a0259f2d788348741df0f7baf167814a02bf41ae30fa6aa"}
Nov 03 08:08:03.202626 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:03.202576    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2" podStartSLOduration=2.202562256 podStartE2EDuration="2.202562256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:03.195850258 +0000 UTC m=+881.310314188" watchObservedRunningTime="2026-11-03 08:08:03.202562256 +0000 UTC m=+881.317026189"
Nov 03 08:08:03.553609 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:03.553525    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 08:08:03.553609 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:03.553586    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6" podUID="7f0c52e6-b384-411f-9c3a-c2d76c149238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 08:08:04.192394 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:04.192359    5070 generic.go:356] "Generic (PLEG): container finished" podID="3a67dfc8-06fa-4be2-9be2-40477be1c36c" containerID="97f795f398132977c14fd2f64b5399a3a3a2f2d7730c616628529acc52ae721f" exitCode=0
Nov 03 08:08:04.192850 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:04.192447    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2" event={"ID":"3a67dfc8-06fa-4be2-9be2-40477be1c36c","Type":"ContainerDied","Data":"97f795f398132977c14fd2f64b5399a3a3a2f2d7730c616628529acc52ae721f"}
Nov 03 08:08:04.552517 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:04.552440    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 08:08:04.552517 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:04.552499    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6" podUID="7f0c52e6-b384-411f-9c3a-c2d76c149238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 08:08:05.446537 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:05.446512    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2"
Nov 03 08:08:05.519581 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:05.519502    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a67dfc8-06fa-4be2-9be2-40477be1c36c-kube-api-access-strnj\" (UniqueName: \"kubernetes.io/projected/3a67dfc8-06fa-4be2-9be2-40477be1c36c-kube-api-access-strnj\") pod \"3a67dfc8-06fa-4be2-9be2-40477be1c36c\" (UID: \"3a67dfc8-06fa-4be2-9be2-40477be1c36c\") "
Nov 03 08:08:05.519855 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:05.519833    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a67dfc8-06fa-4be2-9be2-40477be1c36c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a67dfc8-06fa-4be2-9be2-40477be1c36c-maas-api-xks-ca\") pod \"3a67dfc8-06fa-4be2-9be2-40477be1c36c\" (UID: \"3a67dfc8-06fa-4be2-9be2-40477be1c36c\") "
Nov 03 08:08:05.520085 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:05.520066    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a67dfc8-06fa-4be2-9be2-40477be1c36c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a67dfc8-06fa-4be2-9be2-40477be1c36c-maas-api-openshift-ca\") pod \"3a67dfc8-06fa-4be2-9be2-40477be1c36c\" (UID: \"3a67dfc8-06fa-4be2-9be2-40477be1c36c\") "
Nov 03 08:08:05.521445 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:05.521421    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a67dfc8-06fa-4be2-9be2-40477be1c36c-maas-api-openshift-ca" pod "3a67dfc8-06fa-4be2-9be2-40477be1c36c" (UID: "3a67dfc8-06fa-4be2-9be2-40477be1c36c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:08:05.524311 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:05.524273    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a67dfc8-06fa-4be2-9be2-40477be1c36c-maas-api-xks-ca" pod "3a67dfc8-06fa-4be2-9be2-40477be1c36c" (UID: "3a67dfc8-06fa-4be2-9be2-40477be1c36c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:08:05.525967 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:05.525944    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a67dfc8-06fa-4be2-9be2-40477be1c36c-kube-api-access-strnj" pod "3a67dfc8-06fa-4be2-9be2-40477be1c36c" (UID: "3a67dfc8-06fa-4be2-9be2-40477be1c36c"). InnerVolumeSpecName "kube-api-access-strnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:08:05.552662 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:05.552637    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 08:08:05.552787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:05.552683    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6" podUID="7f0c52e6-b384-411f-9c3a-c2d76c149238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 08:08:05.622586 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:05.622553    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-strnj\" (UniqueName: \"kubernetes.io/projected/3a67dfc8-06fa-4be2-9be2-40477be1c36c-kube-api-access-strnj\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:08:05.622586 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:05.622585    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a67dfc8-06fa-4be2-9be2-40477be1c36c-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:08:05.622801 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:05.622597    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a67dfc8-06fa-4be2-9be2-40477be1c36c-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:08:06.212205 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:06.212178    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2"
Nov 03 08:08:06.212205 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:06.212189    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2" event={"ID":"3a67dfc8-06fa-4be2-9be2-40477be1c36c","Type":"ContainerDied","Data":"4104cee45e0196235a0259f2d788348741df0f7baf167814a02bf41ae30fa6aa"}
Nov 03 08:08:06.212414 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:06.212247    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4104cee45e0196235a0259f2d788348741df0f7baf167814a02bf41ae30fa6aa"
Nov 03 08:08:06.556374 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:06.555885    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 08:08:06.556374 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:06.555972    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6" podUID="7f0c52e6-b384-411f-9c3a-c2d76c149238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 08:08:07.552683 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:07.552646    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 08:08:07.552914 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:07.552710    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6" podUID="7f0c52e6-b384-411f-9c3a-c2d76c149238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 08:08:08.553383 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:08.553341    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 08:08:08.553752 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:08.553431    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6" podUID="7f0c52e6-b384-411f-9c3a-c2d76c149238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 08:08:09.779169 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:09.779137    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:09.779654 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:09.779632    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:09.780452 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:09.780429    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:08:12.464097 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:12.464051    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:12.477782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:12.477727    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:08:12.946914 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:12.946859    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:12.962897 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:12.962846    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:13.079910 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:13.079858    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:13.092975 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:13.092934    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:13.164909 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:13.164836    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:08:13.284181 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:13.284081    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:13.572835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:13.478042    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:13.634336 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:13.486968    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:08:13.647465 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:13.487015    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:13.648088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:13.648058    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:08:13.770942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:13.769867    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:08:13.845924 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:13.845822    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292"]
Nov 03 08:08:13.882781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:13.847263    5070 cpu_manager.go:414] "containerMap: removing container" podUID="3a67dfc8-06fa-4be2-9be2-40477be1c36c" containerName="revoke-keys"
Nov 03 08:08:13.883089 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:13.883047    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a67dfc8-06fa-4be2-9be2-40477be1c36c" containerName="revoke-keys"
Nov 03 08:08:13.883248 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:13.883234    5070 cpu_manager.go:414] "containerMap: removing container" podUID="e1653421-2455-40e3-9103-f512bb881f04" containerName="revoke-keys"
Nov 03 08:08:13.895861 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:13.895830    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1653421-2455-40e3-9103-f512bb881f04" containerName="revoke-keys"
Nov 03 08:08:13.970913 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:13.970863    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:08:14.013787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:13.974016    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:08:14.077783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.058314    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:14.082901 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.082850    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:14.121076 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.108869    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:08:14.129778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.114287    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a67dfc8-06fa-4be2-9be2-40477be1c36c" containerName="revoke-keys"
Nov 03 08:08:14.129778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.126698    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1653421-2455-40e3-9103-f512bb881f04" containerName="revoke-keys"
Nov 03 08:08:14.146925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.146886    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292"
Nov 03 08:08:14.151352 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.147142    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:14.185812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.179330    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:14.267243 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.267196    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0a9dd59a\""
Nov 03 08:08:14.296823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.290907    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:08:14.296823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.295979    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0a9dd59a-dockercfg-55d7l\""
Nov 03 08:08:14.391834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.391701    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlqd9\" (UniqueName: \"kubernetes.io/projected/8f2e9c62-2b7c-452c-8915-05983bc505df-kube-api-access-jlqd9\") pod \"payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292\" (UID: \"8f2e9c62-2b7c-452c-8915-05983bc505df\") " pod="openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292"
Nov 03 08:08:14.486782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.486001    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2e9c62-2b7c-452c-8915-05983bc505df-plugins-config-volume\") pod \"payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292\" (UID: \"8f2e9c62-2b7c-452c-8915-05983bc505df\") " pod="openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292"
Nov 03 08:08:14.606784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.602385    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2e9c62-2b7c-452c-8915-05983bc505df-plugins-config-volume\") pod \"payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292\" (UID: \"8f2e9c62-2b7c-452c-8915-05983bc505df\") " pod="openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292"
Nov 03 08:08:14.606784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.602558    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlqd9\" (UniqueName: \"kubernetes.io/projected/8f2e9c62-2b7c-452c-8915-05983bc505df-kube-api-access-jlqd9\") pod \"payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292\" (UID: \"8f2e9c62-2b7c-452c-8915-05983bc505df\") " pod="openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292"
Nov 03 08:08:14.644838 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.624205    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:08:14.776783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.751987    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:08:14.843889 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.843858    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:08:14.844158 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.844140    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:08:14.844306 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.844294    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:08:14.844429 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.844418    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:08:14.883945 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.735582    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:15.007841 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.645099    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:15.007841 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.735620    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:08:15.007841 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.904524    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292"]
Nov 03 08:08:15.007841 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.986086    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:08:15.007841 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:14.987057    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:08:15.053929 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:15.053895    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2e9c62-2b7c-452c-8915-05983bc505df-plugins-config-volume\") pod \"payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292\" (UID: \"8f2e9c62-2b7c-452c-8915-05983bc505df\") " pod="openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292"
Nov 03 08:08:15.104780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:15.097483    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlqd9\" (UniqueName: \"kubernetes.io/projected/8f2e9c62-2b7c-452c-8915-05983bc505df-kube-api-access-jlqd9\") pod \"payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292\" (UID: \"8f2e9c62-2b7c-452c-8915-05983bc505df\") " pod="openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292"
Nov 03 08:08:15.109781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:15.107900    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:08:15.113781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:15.111138    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:15.138887 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:15.138837    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:08:15.279088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:15.278982    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292"
Nov 03 08:08:15.291069 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:15.279370    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:08:15.375584 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:15.375544    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:08:15.434797 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:15.425568    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:16.021236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:16.021134    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:16.023372 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:16.023327    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:16.024543 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:16.024515    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:08:16.025732 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:16.025708    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:08:16.117855 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:16.117807    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:16.145785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:16.134899    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:16.145785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:16.141286    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:08:16.293288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:16.293192    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:08:16.408813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:16.403305    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:16.510220 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:16.510176    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Nov 03 08:08:16.693998 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:16.693934    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4" podUID="d50d86f3-08a1-419c-a77b-d7223e0567c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:16.814956 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:16.814911    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:16.815492 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:16.815461    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:08:16.815683 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:16.815660    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr" podUID="64c976ca-8c6d-4835-b5ca-ba88be627ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:17.025852 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:17.023510    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:17.054557 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:17.054500    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:08:17.096694 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:17.062525    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:08:17.114784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:17.062685    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:08:17.136209 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:17.136160    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:17.137534 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:17.137497    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:08:17.147466 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:08:17.147433    5070 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.08s"
Nov 03 08:08:17.306102 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:17.284574    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:08:17.475198 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:17.471784    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:08:17.872925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:17.476209    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:08:18.023837 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:18.023793    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:18.309835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:18.283726    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:18.309835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:18.284102    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:08:18.309835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:18.284349    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:08:18.329918 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:18.329875    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:08:18.385088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:18.385046    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:08:18.426732 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:18.426679    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:18.496784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:18.489201    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:18.634841 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:18.615058    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:18.634841 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:18.625863    5070 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-j5h2j container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:08:18.634841 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:18.625946    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j" podUID="22591a72-cf89-4406-8e20-82fd6d0f6e6a" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:18.641135 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:18.641031    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j" podUID="67eb2350-8895-4e7d-b1f2-e78e1ff5cf23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:19.000855 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:18.994907    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw" podUID="0d26093c-a60a-4cd8-9d43-8ecded62d895" containerName="server" probeResult="failure" output="Get \"https://10.132.0.47:8443/healthz\": context deadline exceeded"
Nov 03 08:08:19.024919 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:19.024859    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:19.330421 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:19.330314    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:19.332180 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:19.332132    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:19.333426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:19.333392    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:19.346530 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:19.346501    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded"
Nov 03 08:08:19.347001 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:19.346968    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:19.478782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:19.464704    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:08:19.478782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:19.465415    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:19.497853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:19.496879    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:19.840807 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:19.839153    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:08:19.927197 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:19.927110    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw"]
Nov 03 08:08:19.938825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:19.938798    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw"
Nov 03 08:08:19.965883 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:19.951353    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:08:19.965883 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:19.961241    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:08:19.965883 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:19.961326    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:08:19.966588 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:19.966549    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:08:20.100458 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.099792    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c7bcaf9-f7f9-429e-821c-ade13425d122-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7e28100-v9xvw\" (UID: \"7c7bcaf9-f7f9-429e-821c-ade13425d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw"
Nov 03 08:08:20.100458 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.100103    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6tjd\" (UniqueName: \"kubernetes.io/projected/7c7bcaf9-f7f9-429e-821c-ade13425d122-kube-api-access-v6tjd\") pod \"maas-api-revoke-sub-e7e28100-v9xvw\" (UID: \"7c7bcaf9-f7f9-429e-821c-ade13425d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw"
Nov 03 08:08:20.100458 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.100266    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c7bcaf9-f7f9-429e-821c-ade13425d122-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7e28100-v9xvw\" (UID: \"7c7bcaf9-f7f9-429e-821c-ade13425d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw"
Nov 03 08:08:20.226829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.163828    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": context deadline exceeded"
Nov 03 08:08:20.226829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.164087    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:20.226829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.203348    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6tjd\" (UniqueName: \"kubernetes.io/projected/7c7bcaf9-f7f9-429e-821c-ade13425d122-kube-api-access-v6tjd\") pod \"maas-api-revoke-sub-e7e28100-v9xvw\" (UID: \"7c7bcaf9-f7f9-429e-821c-ade13425d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw"
Nov 03 08:08:20.226829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.203505    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c7bcaf9-f7f9-429e-821c-ade13425d122-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7e28100-v9xvw\" (UID: \"7c7bcaf9-f7f9-429e-821c-ade13425d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw"
Nov 03 08:08:20.226829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.203614    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c7bcaf9-f7f9-429e-821c-ade13425d122-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7e28100-v9xvw\" (UID: \"7c7bcaf9-f7f9-429e-821c-ade13425d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw"
Nov 03 08:08:20.226829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.218094    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c7bcaf9-f7f9-429e-821c-ade13425d122-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7e28100-v9xvw\" (UID: \"7c7bcaf9-f7f9-429e-821c-ade13425d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw"
Nov 03 08:08:20.343089 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.337453    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:08:20.351361 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.344090    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c7bcaf9-f7f9-429e-821c-ade13425d122-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7e28100-v9xvw\" (UID: \"7c7bcaf9-f7f9-429e-821c-ade13425d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw"
Nov 03 08:08:20.351361 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.350436    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:20.357814 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.356721    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:08:20.357814 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.356831    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw"]
Nov 03 08:08:20.382819 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.376862    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:20.469922 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.469883    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:20.470523 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.470493    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:20.532130 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.498849    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:08:20.574694 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.574655    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6tjd\" (UniqueName: \"kubernetes.io/projected/7c7bcaf9-f7f9-429e-821c-ade13425d122-kube-api-access-v6tjd\") pod \"maas-api-revoke-sub-e7e28100-v9xvw\" (UID: \"7c7bcaf9-f7f9-429e-821c-ade13425d122\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw"
Nov 03 08:08:20.578486 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.578455    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:08:20.581291 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.581072    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:08:20.582989 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.582533    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:08:20.583936 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.583684    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:08:20.584495 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.584477    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:08:20.586382 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.586365    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:08:20.592741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.592717    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:08:20.595264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.595242    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292" event={"ID":"8f2e9c62-2b7c-452c-8915-05983bc505df","Type":"ContainerStarted","Data":"54f24bd56185742af4831862b81e8b2e9f1cd27035f77107c9d2e1c92b5e9d23"}
Nov 03 08:08:20.595414 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.595399    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292" event={"ID":"8f2e9c62-2b7c-452c-8915-05983bc505df","Type":"ContainerStarted","Data":"46f3fae82718ed2244d9fd62d0ae551d08919745b05119fdc85a595c6a46b18f"}
Nov 03 08:08:20.600112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.597515    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292"
Nov 03 08:08:20.645724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.645681    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"]
Nov 03 08:08:20.657035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.657006    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"]
Nov 03 08:08:20.657281 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.657258    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.662973 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.660879    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292"]
Nov 03 08:08:20.700660 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.700628    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw"
Nov 03 08:08:20.706020 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.705942    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.706187 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.706062    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.706187 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.706152    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.706374 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.706209    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1cb7af3-82c2-482c-aa2d-510c50735024-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.706374 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.706241    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.707003 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.706917    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.707181 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.707009    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.707272 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.707255    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nf9m\" (UniqueName: \"kubernetes.io/projected/b1cb7af3-82c2-482c-aa2d-510c50735024-kube-api-access-4nf9m\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.707371 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.707327    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.743319 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.743264    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292" podStartSLOduration=7.74324619 podStartE2EDuration="7.74324619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:20.65960126 +0000 UTC m=+898.774065194" watchObservedRunningTime="2026-11-03 08:08:20.74324619 +0000 UTC m=+898.857710124"
Nov 03 08:08:20.810849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.809475    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nf9m\" (UniqueName: \"kubernetes.io/projected/b1cb7af3-82c2-482c-aa2d-510c50735024-kube-api-access-4nf9m\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.810849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.809616    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.810849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.809814    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.810849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.809923    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.810849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.809999    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.810849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.810043    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1cb7af3-82c2-482c-aa2d-510c50735024-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.810849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.810081    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.810849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.810119    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.810849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.810156    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.812043 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.812017    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-data\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.812194 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.812090    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1cb7af3-82c2-482c-aa2d-510c50735024-istiod-ca-cert\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.812370 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.812347    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-workload-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.812715 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.812696    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-workload-certs\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.813340 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.813320    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-credential-socket\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.829289 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.829259    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-podinfo\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.829745 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.829683    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nf9m\" (UniqueName: \"kubernetes.io/projected/b1cb7af3-82c2-482c-aa2d-510c50735024-kube-api-access-4nf9m\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.830490 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.830258    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-token\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:20.831034 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:20.831009    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-envoy\") pod \"e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") " pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:21.002466 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:21.002408    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:21.026804 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:21.026723    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw"]
Nov 03 08:08:21.031458 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:08:21.031423    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7bcaf9_f7f9_429e_821c_ade13425d122.slice/crio-4a34cd020d56802f9624376945565c76b3a28503311b2cfa866b38b2dd703099 WatchSource:0}: Error finding container 4a34cd020d56802f9624376945565c76b3a28503311b2cfa866b38b2dd703099: Status 404 returned error can't find the container with id 4a34cd020d56802f9624376945565c76b3a28503311b2cfa866b38b2dd703099
Nov 03 08:08:21.321245 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:21.321217    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"]
Nov 03 08:08:21.321393 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:21.321322    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:08:21.321393 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:21.321382    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:08:21.321506 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:21.321413    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:08:21.606896 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:21.606462    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q" event={"ID":"b1cb7af3-82c2-482c-aa2d-510c50735024","Type":"ContainerStarted","Data":"2e46fc0ae212af5ae9cfd6003a9203d3f9bc74178aeb56ad9dbc7926011bbb5c"}
Nov 03 08:08:21.606896 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:21.606524    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q" event={"ID":"b1cb7af3-82c2-482c-aa2d-510c50735024","Type":"ContainerStarted","Data":"5b51fda80ac1b6d422a0fdea95cba9a3493c298ee51393a15e091b7979b0f06a"}
Nov 03 08:08:21.611187 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:21.611084    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw" event={"ID":"7c7bcaf9-f7f9-429e-821c-ade13425d122","Type":"ContainerStarted","Data":"8dda71f828b1aa827dbf5f149fefffcb7f07b97d7a0a8d122039295a2d8ecdce"}
Nov 03 08:08:21.611187 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:21.611136    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw" event={"ID":"7c7bcaf9-f7f9-429e-821c-ade13425d122","Type":"ContainerStarted","Data":"4a34cd020d56802f9624376945565c76b3a28503311b2cfa866b38b2dd703099"}
Nov 03 08:08:21.659838 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:21.659779    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw" podStartSLOduration=6.65974939 podStartE2EDuration="6.65974939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:21.657883608 +0000 UTC m=+899.772347567" watchObservedRunningTime="2026-11-03 08:08:21.65974939 +0000 UTC m=+899.774213321"
Nov 03 08:08:21.664960 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:21.664920    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q" podStartSLOduration=6.664907358 podStartE2EDuration="6.664907358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:21.64068185 +0000 UTC m=+899.755145807" watchObservedRunningTime="2026-11-03 08:08:21.664907358 +0000 UTC m=+899.779371289"
Nov 03 08:08:22.002830 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:22.002750    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:22.004194 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:22.004162    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 03 08:08:22.004298 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:22.004223    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q" podUID="b1cb7af3-82c2-482c-aa2d-510c50735024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 03 08:08:22.068452 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:08:22.068417    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:08:22.244090 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:22.244059    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/2.log"
Nov 03 08:08:22.244260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:22.244063    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/2.log"
Nov 03 08:08:22.621568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:22.621533    5070 generic.go:356] "Generic (PLEG): container finished" podID="7c7bcaf9-f7f9-429e-821c-ade13425d122" containerID="8dda71f828b1aa827dbf5f149fefffcb7f07b97d7a0a8d122039295a2d8ecdce" exitCode=0
Nov 03 08:08:22.622103 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:22.621614    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw" event={"ID":"7c7bcaf9-f7f9-429e-821c-ade13425d122","Type":"ContainerDied","Data":"8dda71f828b1aa827dbf5f149fefffcb7f07b97d7a0a8d122039295a2d8ecdce"}
Nov 03 08:08:23.003768 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:23.003672    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 03 08:08:23.003768 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:23.003734    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q" podUID="b1cb7af3-82c2-482c-aa2d-510c50735024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 03 08:08:23.886967 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:23.886901    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw"
Nov 03 08:08:23.909724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:23.909688    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c7bcaf9-f7f9-429e-821c-ade13425d122-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c7bcaf9-f7f9-429e-821c-ade13425d122-maas-api-xks-ca\") pod \"7c7bcaf9-f7f9-429e-821c-ade13425d122\" (UID: \"7c7bcaf9-f7f9-429e-821c-ade13425d122\") "
Nov 03 08:08:23.910021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:23.909998    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c7bcaf9-f7f9-429e-821c-ade13425d122-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c7bcaf9-f7f9-429e-821c-ade13425d122-maas-api-openshift-ca\") pod \"7c7bcaf9-f7f9-429e-821c-ade13425d122\" (UID: \"7c7bcaf9-f7f9-429e-821c-ade13425d122\") "
Nov 03 08:08:23.910135 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:23.910072    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7bcaf9-f7f9-429e-821c-ade13425d122-kube-api-access-v6tjd\" (UniqueName: \"kubernetes.io/projected/7c7bcaf9-f7f9-429e-821c-ade13425d122-kube-api-access-v6tjd\") pod \"7c7bcaf9-f7f9-429e-821c-ade13425d122\" (UID: \"7c7bcaf9-f7f9-429e-821c-ade13425d122\") "
Nov 03 08:08:23.910391 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:23.910339    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7bcaf9-f7f9-429e-821c-ade13425d122-maas-api-openshift-ca" pod "7c7bcaf9-f7f9-429e-821c-ade13425d122" (UID: "7c7bcaf9-f7f9-429e-821c-ade13425d122"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:08:23.910796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:23.910738    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c7bcaf9-f7f9-429e-821c-ade13425d122-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:08:23.911945 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:23.911921    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c7bcaf9-f7f9-429e-821c-ade13425d122-maas-api-xks-ca" pod "7c7bcaf9-f7f9-429e-821c-ade13425d122" (UID: "7c7bcaf9-f7f9-429e-821c-ade13425d122"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:08:23.912475 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:23.912449    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7bcaf9-f7f9-429e-821c-ade13425d122-kube-api-access-v6tjd" pod "7c7bcaf9-f7f9-429e-821c-ade13425d122" (UID: "7c7bcaf9-f7f9-429e-821c-ade13425d122"). InnerVolumeSpecName "kube-api-access-v6tjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:08:24.036065 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:24.029168    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 03 08:08:24.036065 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:24.029225    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q" podUID="b1cb7af3-82c2-482c-aa2d-510c50735024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 03 08:08:24.040071 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:24.039264    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6tjd\" (UniqueName: \"kubernetes.io/projected/7c7bcaf9-f7f9-429e-821c-ade13425d122-kube-api-access-v6tjd\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:08:24.040071 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:24.039294    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c7bcaf9-f7f9-429e-821c-ade13425d122-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:08:24.523788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:24.523744    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb"]
Nov 03 08:08:24.524704 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:24.524677    5070 cpu_manager.go:414] "containerMap: removing container" podUID="7c7bcaf9-f7f9-429e-821c-ade13425d122" containerName="revoke-keys"
Nov 03 08:08:24.524704 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:24.524699    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7bcaf9-f7f9-429e-821c-ade13425d122" containerName="revoke-keys"
Nov 03 08:08:24.524967 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:24.524947    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7bcaf9-f7f9-429e-821c-ade13425d122" containerName="revoke-keys"
Nov 03 08:08:24.528684 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:24.528666    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb"
Nov 03 08:08:24.537658 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:24.531697    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-0b397f45-dockercfg-ws6nw\""
Nov 03 08:08:24.537658 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:24.532498    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-0b397f45\""
Nov 03 08:08:24.588412 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:24.588376    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb"]
Nov 03 08:08:24.752166 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:24.751455    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5q4n\" (UniqueName: \"kubernetes.io/projected/8174368c-689f-4bb0-a8a8-3a58f8916807-kube-api-access-b5q4n\") pod \"payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb\" (UID: \"8174368c-689f-4bb0-a8a8-3a58f8916807\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb"
Nov 03 08:08:24.752166 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:24.751616    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8174368c-689f-4bb0-a8a8-3a58f8916807-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb\" (UID: \"8174368c-689f-4bb0-a8a8-3a58f8916807\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb"
Nov 03 08:08:24.862850 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:24.860722    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5q4n\" (UniqueName: \"kubernetes.io/projected/8174368c-689f-4bb0-a8a8-3a58f8916807-kube-api-access-b5q4n\") pod \"payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb\" (UID: \"8174368c-689f-4bb0-a8a8-3a58f8916807\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb"
Nov 03 08:08:24.862850 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:24.860878    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8174368c-689f-4bb0-a8a8-3a58f8916807-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb\" (UID: \"8174368c-689f-4bb0-a8a8-3a58f8916807\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb"
Nov 03 08:08:24.862850 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:24.861910    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8174368c-689f-4bb0-a8a8-3a58f8916807-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb\" (UID: \"8174368c-689f-4bb0-a8a8-3a58f8916807\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb"
Nov 03 08:08:24.897892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:24.897512    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw" event={"ID":"7c7bcaf9-f7f9-429e-821c-ade13425d122","Type":"ContainerDied","Data":"4a34cd020d56802f9624376945565c76b3a28503311b2cfa866b38b2dd703099"}
Nov 03 08:08:24.897892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:24.897587    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a34cd020d56802f9624376945565c76b3a28503311b2cfa866b38b2dd703099"
Nov 03 08:08:24.897892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:24.897696    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw"
Nov 03 08:08:24.933920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:24.931870    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5q4n\" (UniqueName: \"kubernetes.io/projected/8174368c-689f-4bb0-a8a8-3a58f8916807-kube-api-access-b5q4n\") pod \"payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb\" (UID: \"8174368c-689f-4bb0-a8a8-3a58f8916807\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb"
Nov 03 08:08:25.004008 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.003965    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 03 08:08:25.004168 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.004044    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q" podUID="b1cb7af3-82c2-482c-aa2d-510c50735024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 03 08:08:25.116839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.116716    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt"]
Nov 03 08:08:25.122068 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.122041    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt"
Nov 03 08:08:25.125323 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.125292    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:08:25.125456 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.125387    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:08:25.136017 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.135997    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt"]
Nov 03 08:08:25.195498 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.195463    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb"
Nov 03 08:08:25.196152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.196134    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:08:25.196334 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.196316    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:08:25.292960 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.288783    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck582\" (UniqueName: \"kubernetes.io/projected/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7-kube-api-access-ck582\") pod \"maas-api-revoke-sub-891bfe36-c2mjt\" (UID: \"9c7858ef-04ee-4155-a69b-b3c3a11e4ea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt"
Nov 03 08:08:25.292960 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.288896    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-891bfe36-c2mjt\" (UID: \"9c7858ef-04ee-4155-a69b-b3c3a11e4ea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt"
Nov 03 08:08:25.292960 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.288998    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-891bfe36-c2mjt\" (UID: \"9c7858ef-04ee-4155-a69b-b3c3a11e4ea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt"
Nov 03 08:08:25.390846 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.390749    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-891bfe36-c2mjt\" (UID: \"9c7858ef-04ee-4155-a69b-b3c3a11e4ea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt"
Nov 03 08:08:25.391066 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.391048    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck582\" (UniqueName: \"kubernetes.io/projected/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7-kube-api-access-ck582\") pod \"maas-api-revoke-sub-891bfe36-c2mjt\" (UID: \"9c7858ef-04ee-4155-a69b-b3c3a11e4ea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt"
Nov 03 08:08:25.391530 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.391138    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-891bfe36-c2mjt\" (UID: \"9c7858ef-04ee-4155-a69b-b3c3a11e4ea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt"
Nov 03 08:08:25.392405 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.392374    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-891bfe36-c2mjt\" (UID: \"9c7858ef-04ee-4155-a69b-b3c3a11e4ea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt"
Nov 03 08:08:25.397511 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.397479    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-891bfe36-c2mjt\" (UID: \"9c7858ef-04ee-4155-a69b-b3c3a11e4ea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt"
Nov 03 08:08:25.404607 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.404586    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck582\" (UniqueName: \"kubernetes.io/projected/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7-kube-api-access-ck582\") pod \"maas-api-revoke-sub-891bfe36-c2mjt\" (UID: \"9c7858ef-04ee-4155-a69b-b3c3a11e4ea7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt"
Nov 03 08:08:25.445113 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.445089    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt"
Nov 03 08:08:25.515513 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.515474    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb"]
Nov 03 08:08:25.515914 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:08:25.515887    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8174368c_689f_4bb0_a8a8_3a58f8916807.slice/crio-87834386469c89cf8a981636797a5f844df9c4af1cb5b904c2b70c208c2a097f WatchSource:0}: Error finding container 87834386469c89cf8a981636797a5f844df9c4af1cb5b904c2b70c208c2a097f: Status 404 returned error can't find the container with id 87834386469c89cf8a981636797a5f844df9c4af1cb5b904c2b70c208c2a097f
Nov 03 08:08:25.885571 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:08:25.885536    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c7858ef_04ee_4155_a69b_b3c3a11e4ea7.slice/crio-3c7ef1b1876e3c99eb4703566d78d2991d1bb0f625bd2230cdfb87ad184084e0 WatchSource:0}: Error finding container 3c7ef1b1876e3c99eb4703566d78d2991d1bb0f625bd2230cdfb87ad184084e0: Status 404 returned error can't find the container with id 3c7ef1b1876e3c99eb4703566d78d2991d1bb0f625bd2230cdfb87ad184084e0
Nov 03 08:08:25.888691 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.888661    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt"]
Nov 03 08:08:25.911380 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.911347    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb" event={"ID":"8174368c-689f-4bb0-a8a8-3a58f8916807","Type":"ContainerStarted","Data":"1fcbada4ad1725ad74ed8a72eefcf44f72cb6eeeb8721797539cf0c25c98404d"}
Nov 03 08:08:25.911673 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.911423    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb" event={"ID":"8174368c-689f-4bb0-a8a8-3a58f8916807","Type":"ContainerStarted","Data":"87834386469c89cf8a981636797a5f844df9c4af1cb5b904c2b70c208c2a097f"}
Nov 03 08:08:25.911673 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.911455    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb"
Nov 03 08:08:25.914099 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.914072    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt" event={"ID":"9c7858ef-04ee-4155-a69b-b3c3a11e4ea7","Type":"ContainerStarted","Data":"3c7ef1b1876e3c99eb4703566d78d2991d1bb0f625bd2230cdfb87ad184084e0"}
Nov 03 08:08:25.928856 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:25.928809    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb" podStartSLOduration=1.928790351 podStartE2EDuration="1.928790351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:25.9243107 +0000 UTC m=+904.038774631" watchObservedRunningTime="2026-11-03 08:08:25.928790351 +0000 UTC m=+904.043254273"
Nov 03 08:08:26.003376 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:26.003191    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 03 08:08:26.003376 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:26.003242    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q" podUID="b1cb7af3-82c2-482c-aa2d-510c50735024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 03 08:08:26.926982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:26.926944    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt" event={"ID":"9c7858ef-04ee-4155-a69b-b3c3a11e4ea7","Type":"ContainerStarted","Data":"0b6bf82e878d3da781dac8d1096c51546c81a109f7913262b3016a94b6d1d049"}
Nov 03 08:08:26.942044 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:26.941992    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt" podStartSLOduration=1.941977446 podStartE2EDuration="1.941977446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:26.939017869 +0000 UTC m=+905.053481800" watchObservedRunningTime="2026-11-03 08:08:26.941977446 +0000 UTC m=+905.056441399"
Nov 03 08:08:27.003096 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:27.003065    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 03 08:08:27.003269 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:27.003111    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q" podUID="b1cb7af3-82c2-482c-aa2d-510c50735024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 03 08:08:27.938236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:27.938202    5070 generic.go:356] "Generic (PLEG): container finished" podID="9c7858ef-04ee-4155-a69b-b3c3a11e4ea7" containerID="0b6bf82e878d3da781dac8d1096c51546c81a109f7913262b3016a94b6d1d049" exitCode=0
Nov 03 08:08:27.938626 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:27.938278    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt" event={"ID":"9c7858ef-04ee-4155-a69b-b3c3a11e4ea7","Type":"ContainerDied","Data":"0b6bf82e878d3da781dac8d1096c51546c81a109f7913262b3016a94b6d1d049"}
Nov 03 08:08:28.004723 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:28.004689    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body=
Nov 03 08:08:28.004915 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:28.004746    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q" podUID="b1cb7af3-82c2-482c-aa2d-510c50735024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused"
Nov 03 08:08:29.158541 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:29.158514    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt"
Nov 03 08:08:29.277391 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:29.277307    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7-maas-api-xks-ca\") pod \"9c7858ef-04ee-4155-a69b-b3c3a11e4ea7\" (UID: \"9c7858ef-04ee-4155-a69b-b3c3a11e4ea7\") "
Nov 03 08:08:29.277552 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:29.277424    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7-kube-api-access-ck582\" (UniqueName: \"kubernetes.io/projected/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7-kube-api-access-ck582\") pod \"9c7858ef-04ee-4155-a69b-b3c3a11e4ea7\" (UID: \"9c7858ef-04ee-4155-a69b-b3c3a11e4ea7\") "
Nov 03 08:08:29.277552 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:29.277457    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7-maas-api-openshift-ca\") pod \"9c7858ef-04ee-4155-a69b-b3c3a11e4ea7\" (UID: \"9c7858ef-04ee-4155-a69b-b3c3a11e4ea7\") "
Nov 03 08:08:29.277965 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:29.277934    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7-maas-api-openshift-ca" pod "9c7858ef-04ee-4155-a69b-b3c3a11e4ea7" (UID: "9c7858ef-04ee-4155-a69b-b3c3a11e4ea7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:08:29.278335 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:29.278303    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:08:29.282914 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:29.282880    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7-maas-api-xks-ca" pod "9c7858ef-04ee-4155-a69b-b3c3a11e4ea7" (UID: "9c7858ef-04ee-4155-a69b-b3c3a11e4ea7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:08:29.283186 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:29.283164    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7-kube-api-access-ck582" pod "9c7858ef-04ee-4155-a69b-b3c3a11e4ea7" (UID: "9c7858ef-04ee-4155-a69b-b3c3a11e4ea7"). InnerVolumeSpecName "kube-api-access-ck582". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:08:29.381573 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:29.381531    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:08:29.381573 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:29.381575    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck582\" (UniqueName: \"kubernetes.io/projected/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7-kube-api-access-ck582\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:08:29.434854 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:29.433672    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:29.436336 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:29.435509    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:29.438827 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:29.437545    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:08:29.963103 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:29.963068    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt"
Nov 03 08:08:29.963103 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:29.963088    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt" event={"ID":"9c7858ef-04ee-4155-a69b-b3c3a11e4ea7","Type":"ContainerDied","Data":"3c7ef1b1876e3c99eb4703566d78d2991d1bb0f625bd2230cdfb87ad184084e0"}
Nov 03 08:08:29.963380 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:29.963133    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c7ef1b1876e3c99eb4703566d78d2991d1bb0f625bd2230cdfb87ad184084e0"
Nov 03 08:08:32.625142 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:32.625113    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292"
Nov 03 08:08:36.930065 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:36.930033    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb"
Nov 03 08:08:46.345728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:46.345685    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:08:46.806747 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:46.806709    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx"]
Nov 03 08:08:46.807522 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:46.807503    5070 cpu_manager.go:414] "containerMap: removing container" podUID="9c7858ef-04ee-4155-a69b-b3c3a11e4ea7" containerName="revoke-keys"
Nov 03 08:08:46.807522 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:46.807522    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c7858ef-04ee-4155-a69b-b3c3a11e4ea7" containerName="revoke-keys"
Nov 03 08:08:46.807701 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:46.807689    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c7858ef-04ee-4155-a69b-b3c3a11e4ea7" containerName="revoke-keys"
Nov 03 08:08:46.812018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:46.811993    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx"
Nov 03 08:08:46.815189 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:46.815163    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:08:46.815316 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:46.815230    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:08:46.815316 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:46.815312    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:08:46.815456 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:46.815410    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:08:46.822381 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:46.822358    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx"]
Nov 03 08:08:46.866982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:46.866749    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd57aa0-260a-4700-a598-da0140f60548-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx\" (UID: \"4fd57aa0-260a-4700-a598-da0140f60548\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx"
Nov 03 08:08:46.867160 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:46.867040    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd57aa0-260a-4700-a598-da0140f60548-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx\" (UID: \"4fd57aa0-260a-4700-a598-da0140f60548\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx"
Nov 03 08:08:46.867234 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:46.867190    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8cqh\" (UniqueName: \"kubernetes.io/projected/4fd57aa0-260a-4700-a598-da0140f60548-kube-api-access-l8cqh\") pod \"maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx\" (UID: \"4fd57aa0-260a-4700-a598-da0140f60548\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx"
Nov 03 08:08:46.968649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:46.968615    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd57aa0-260a-4700-a598-da0140f60548-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx\" (UID: \"4fd57aa0-260a-4700-a598-da0140f60548\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx"
Nov 03 08:08:46.968827 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:46.968723    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8cqh\" (UniqueName: \"kubernetes.io/projected/4fd57aa0-260a-4700-a598-da0140f60548-kube-api-access-l8cqh\") pod \"maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx\" (UID: \"4fd57aa0-260a-4700-a598-da0140f60548\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx"
Nov 03 08:08:46.968895 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:46.968857    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd57aa0-260a-4700-a598-da0140f60548-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx\" (UID: \"4fd57aa0-260a-4700-a598-da0140f60548\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx"
Nov 03 08:08:46.969734 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:46.969703    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd57aa0-260a-4700-a598-da0140f60548-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx\" (UID: \"4fd57aa0-260a-4700-a598-da0140f60548\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx"
Nov 03 08:08:46.971295 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:46.971271    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd57aa0-260a-4700-a598-da0140f60548-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx\" (UID: \"4fd57aa0-260a-4700-a598-da0140f60548\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx"
Nov 03 08:08:47.051617 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:47.051536    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8cqh\" (UniqueName: \"kubernetes.io/projected/4fd57aa0-260a-4700-a598-da0140f60548-kube-api-access-l8cqh\") pod \"maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx\" (UID: \"4fd57aa0-260a-4700-a598-da0140f60548\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx"
Nov 03 08:08:47.131548 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:47.131514    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx"
Nov 03 08:08:47.542831 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:47.542263    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf"]
Nov 03 08:08:47.563276 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:47.563240    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf"
Nov 03 08:08:47.582995 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:47.577020    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf"]
Nov 03 08:08:47.768781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:47.767945    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f832372d-9cd8-466b-a057-5f06e3e372ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf\" (UID: \"f832372d-9cd8-466b-a057-5f06e3e372ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf"
Nov 03 08:08:47.768781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:47.768169    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n84vj\" (UniqueName: \"kubernetes.io/projected/f832372d-9cd8-466b-a057-5f06e3e372ea-kube-api-access-n84vj\") pod \"maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf\" (UID: \"f832372d-9cd8-466b-a057-5f06e3e372ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf"
Nov 03 08:08:47.768781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:47.768229    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f832372d-9cd8-466b-a057-5f06e3e372ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf\" (UID: \"f832372d-9cd8-466b-a057-5f06e3e372ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf"
Nov 03 08:08:47.834786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:47.830914    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx"]
Nov 03 08:08:47.883047 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:47.878311    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f832372d-9cd8-466b-a057-5f06e3e372ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf\" (UID: \"f832372d-9cd8-466b-a057-5f06e3e372ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf"
Nov 03 08:08:47.883047 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:47.878530    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f832372d-9cd8-466b-a057-5f06e3e372ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf\" (UID: \"f832372d-9cd8-466b-a057-5f06e3e372ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf"
Nov 03 08:08:47.883047 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:47.878732    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n84vj\" (UniqueName: \"kubernetes.io/projected/f832372d-9cd8-466b-a057-5f06e3e372ea-kube-api-access-n84vj\") pod \"maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf\" (UID: \"f832372d-9cd8-466b-a057-5f06e3e372ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf"
Nov 03 08:08:47.887787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:47.884324    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f832372d-9cd8-466b-a057-5f06e3e372ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf\" (UID: \"f832372d-9cd8-466b-a057-5f06e3e372ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf"
Nov 03 08:08:47.887787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:47.885020    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f832372d-9cd8-466b-a057-5f06e3e372ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf\" (UID: \"f832372d-9cd8-466b-a057-5f06e3e372ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf"
Nov 03 08:08:47.899819 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:47.899058    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n84vj\" (UniqueName: \"kubernetes.io/projected/f832372d-9cd8-466b-a057-5f06e3e372ea-kube-api-access-n84vj\") pod \"maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf\" (UID: \"f832372d-9cd8-466b-a057-5f06e3e372ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf"
Nov 03 08:08:48.234935 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:48.232710    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf"
Nov 03 08:08:48.804896 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:48.802206    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx" event={"ID":"4fd57aa0-260a-4700-a598-da0140f60548","Type":"ContainerStarted","Data":"08c1a26481f6a82d34e11f8f1b44bdc8a1dc8473160a6de4523e1588bd9fabff"}
Nov 03 08:08:48.893731 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:48.893691    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb"]
Nov 03 08:08:49.064928 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:48.993310    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:49.209412 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:49.027135    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:49.261583 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:49.204139    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:49.308104 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:49.308071    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb"
Nov 03 08:08:49.337349 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:49.337259    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:49.382710 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:49.381269    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:49.424917 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:49.424175    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d45d5825-e024-4922-8ab1-ba2d417dd6dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb\" (UID: \"d45d5825-e024-4922-8ab1-ba2d417dd6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb"
Nov 03 08:08:49.424917 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:49.424356    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g7lq\" (UniqueName: \"kubernetes.io/projected/d45d5825-e024-4922-8ab1-ba2d417dd6dc-kube-api-access-2g7lq\") pod \"maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb\" (UID: \"d45d5825-e024-4922-8ab1-ba2d417dd6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb"
Nov 03 08:08:49.424917 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:49.424434    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d45d5825-e024-4922-8ab1-ba2d417dd6dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb\" (UID: \"d45d5825-e024-4922-8ab1-ba2d417dd6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb"
Nov 03 08:08:49.528976 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:49.527586    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d45d5825-e024-4922-8ab1-ba2d417dd6dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb\" (UID: \"d45d5825-e024-4922-8ab1-ba2d417dd6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb"
Nov 03 08:08:49.528976 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:49.527801    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g7lq\" (UniqueName: \"kubernetes.io/projected/d45d5825-e024-4922-8ab1-ba2d417dd6dc-kube-api-access-2g7lq\") pod \"maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb\" (UID: \"d45d5825-e024-4922-8ab1-ba2d417dd6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb"
Nov 03 08:08:49.528976 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:49.527889    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d45d5825-e024-4922-8ab1-ba2d417dd6dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb\" (UID: \"d45d5825-e024-4922-8ab1-ba2d417dd6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb"
Nov 03 08:08:49.533428 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:49.533404    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d45d5825-e024-4922-8ab1-ba2d417dd6dc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb\" (UID: \"d45d5825-e024-4922-8ab1-ba2d417dd6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb"
Nov 03 08:08:49.563890 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:49.563845    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb"]
Nov 03 08:08:49.594827 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:49.584474    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx" podStartSLOduration=3.584452137 podStartE2EDuration="3.584452137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:08:48.839481979 +0000 UTC m=+926.953945914" watchObservedRunningTime="2026-11-03 08:08:49.584452137 +0000 UTC m=+927.698916069"
Nov 03 08:08:49.625402 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:49.625372    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g7lq\" (UniqueName: \"kubernetes.io/projected/d45d5825-e024-4922-8ab1-ba2d417dd6dc-kube-api-access-2g7lq\") pod \"maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb\" (UID: \"d45d5825-e024-4922-8ab1-ba2d417dd6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb"
Nov 03 08:08:49.682171 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:49.681904    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:49.704275 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:49.704208    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d45d5825-e024-4922-8ab1-ba2d417dd6dc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb\" (UID: \"d45d5825-e024-4922-8ab1-ba2d417dd6dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb"
Nov 03 08:08:49.748187 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:49.748155    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb"
Nov 03 08:08:50.127788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:50.093948    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:08:50.127788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:50.094037    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt" podUID="f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:50.292147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:50.262933    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:50.310051 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:50.310000    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:50.451946 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:50.310849    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:50.451946 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:50.374248    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:50.451946 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:50.427449    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:08:50.472948 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:50.472036    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:50.704010 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:50.567295    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:08:50.704010 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:50.641917    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:08:51.088789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:51.085692    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:51.105793 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:50.907419    5070 generic.go:356] "Generic (PLEG): container finished" podID="4fd57aa0-260a-4700-a598-da0140f60548" containerID="d0bb89b21273c681c2a9cb8d55c362a333f6e42ac96a8a0b5485aa9aa1202529" exitCode=0
Nov 03 08:08:51.605795 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:51.602353    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:08:51.630783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:51.606868    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:08:51.630783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:51.608194    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:08:51.630783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:51.623646    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:08:51.722799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:51.636368    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:08:51.722799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:51.642623    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:51.722799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:51.657664    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:08:51.722799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:51.667212    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:51.722799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:51.677415    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:51.722799 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:08:51.701563    5070 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.652s"
Nov 03 08:08:51.722799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:51.701742    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:08:51.722799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:51.701803    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:08:51.722799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:51.701839    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:08:51.722799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:51.701872    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:08:51.722799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:51.701905    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:08:51.722799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:51.701921    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx" event={"ID":"4fd57aa0-260a-4700-a598-da0140f60548","Type":"ContainerDied","Data":"d0bb89b21273c681c2a9cb8d55c362a333f6e42ac96a8a0b5485aa9aa1202529"}
Nov 03 08:08:51.722799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:51.701975    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:08:51.722799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:51.702010    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:08:51.722799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:51.702043    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:08:51.783379 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:51.702871    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz" podUID="33f324bd-79b1-4540-a090-7e4a9d92f10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:52.213835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:52.202379    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:52.682786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:52.670435    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:52.790258 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:52.670602    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:08:52.791300 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:52.670636    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:08:52.814632 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:52.743107    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:53.311873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:52.878494    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:53.311873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:52.910956    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:53.311873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:52.946879    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:08:53.311873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:52.946936    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:08:53.311873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:53.173433    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:53.379216 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:08:53.379179    5070 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.219s"
Nov 03 08:08:53.379508 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:53.379488    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:08:53.379654 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:53.379640    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf"]
Nov 03 08:08:53.402838 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:53.382077    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf" event={"ID":"f832372d-9cd8-466b-a057-5f06e3e372ea","Type":"ContainerStarted","Data":"5e7db7f9e02b59bf5703ab3817895dfef89e7fdd52d5b2132fe304fb6c59b8b1"}
Nov 03 08:08:53.575226 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:08:53.575123    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:08:53.835282 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:53.813177    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:53.835282 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:53.813511    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:08:53.835282 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:53.813791    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:08:54.259076 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:54.259022    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:54.475440 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:54.430970    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": context deadline exceeded"
Nov 03 08:08:54.475440 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:54.431138    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:08:54.475440 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:54.431587    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:54.475440 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:54.432162    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:08:54.735722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:54.735599    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded"
Nov 03 08:08:54.770969 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:54.770925    5070 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:08:54.771268 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:54.771215    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx" podUID="d7faa6ae-27c1-4a98-8c83-4a3fb1669835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:54.949406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:54.949362    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4" podUID="d50d86f3-08a1-419c-a77b-d7223e0567c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:55.020656 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:55.020562    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:08:55.020946 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:55.020913    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89" podUID="bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:55.118012 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:55.117950    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:55.121151 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:55.121100    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:08:55.121388 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:55.121353    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv" podUID="e724a396-646c-42bb-9b59-d7ff618e07f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:55.170987 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:55.165242    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:08:55.170987 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:55.165313    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd" podUID="4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:55.188278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:55.180556    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:55.220925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:55.220878    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:55.283833 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:55.266858    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:55.353784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:55.322826    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:55.353784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:55.327920    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:08:55.512792 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:55.488892    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:08:55.512792 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:55.488970    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt" podUID="56d6e143-5bb7-42a8-a5e6-0f14214b7592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:55.555479 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:55.542540    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:55.555479 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:55.543011    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:08:55.555479 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:55.543304    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:08:55.823872 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:55.811582    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:08:55.823872 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:55.811668    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs" podUID="2abf1628-8c68-450c-a458-eefa78b239b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:56.086479 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:56.060409    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:08:56.086479 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:56.060469    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6" podUID="7f0c52e6-b384-411f-9c3a-c2d76c149238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:56.140843 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:56.121801    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:08:56.201906 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:56.201852    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:56.289983 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:56.231819    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:56.407001 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:56.406935    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:56.487894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:56.465558    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:56.487894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:56.482422    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:08:56.662079 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:56.650302    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Nov 03 08:08:56.662079 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:56.661039    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:08:56.662079 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:56.661265    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:56.690414 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:56.674860    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": context deadline exceeded"
Nov 03 08:08:56.820785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:56.802954    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:57.002457 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:08:57.000556    5070 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.695s"
Nov 03 08:08:57.125783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:57.122266    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:57.243319 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:57.178918    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded"
Nov 03 08:08:57.292847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:57.277501    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:57.329461 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:57.329420    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:57.427074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:57.427015    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:08:57.430057 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:57.430031    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx"
Nov 03 08:08:57.552353 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:57.552224    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:08:57.558767 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:57.558720    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fd57aa0-260a-4700-a598-da0140f60548-kube-api-access-l8cqh\" (UniqueName: \"kubernetes.io/projected/4fd57aa0-260a-4700-a598-da0140f60548-kube-api-access-l8cqh\") pod \"4fd57aa0-260a-4700-a598-da0140f60548\" (UID: \"4fd57aa0-260a-4700-a598-da0140f60548\") "
Nov 03 08:08:57.567289 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:57.567255    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fd57aa0-260a-4700-a598-da0140f60548-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd57aa0-260a-4700-a598-da0140f60548-maas-api-openshift-ca\") pod \"4fd57aa0-260a-4700-a598-da0140f60548\" (UID: \"4fd57aa0-260a-4700-a598-da0140f60548\") "
Nov 03 08:08:57.567503 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:57.567482    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fd57aa0-260a-4700-a598-da0140f60548-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd57aa0-260a-4700-a598-da0140f60548-maas-api-xks-ca\") pod \"4fd57aa0-260a-4700-a598-da0140f60548\" (UID: \"4fd57aa0-260a-4700-a598-da0140f60548\") "
Nov 03 08:08:57.600269 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:57.600230    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd57aa0-260a-4700-a598-da0140f60548-maas-api-openshift-ca" pod "4fd57aa0-260a-4700-a598-da0140f60548" (UID: "4fd57aa0-260a-4700-a598-da0140f60548"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:08:57.600436 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:57.600336    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:57.671944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:57.664856    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:57.711854 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:57.677559    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd57aa0-260a-4700-a598-da0140f60548-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:08:57.812854 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:57.812415    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:57.917265 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:57.900285    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:08:58.131832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.129904    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:58.236373 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:08:58.218836    5070 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.218s"
Nov 03 08:08:58.236373 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.218874    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb" event={"ID":"d45d5825-e024-4922-8ab1-ba2d417dd6dc","Type":"ContainerStarted","Data":"e2d83dee4ad9c302dcf55c6f34f620c570d8b392a0048f70093af6682cae642b"}
Nov 03 08:08:58.236373 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.219026    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf" event={"ID":"f832372d-9cd8-466b-a057-5f06e3e372ea","Type":"ContainerStarted","Data":"0024023d85a82fa0aec464eb3c50ae65ae41a01be8a38dcd0bb643a2e157dcff"}
Nov 03 08:08:58.287845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.287787    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd57aa0-260a-4700-a598-da0140f60548-maas-api-xks-ca" pod "4fd57aa0-260a-4700-a598-da0140f60548" (UID: "4fd57aa0-260a-4700-a598-da0140f60548"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:08:58.331578 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.330249    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:58.371788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.369650    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd57aa0-260a-4700-a598-da0140f60548-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:08:58.386505 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.372713    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:58.452020 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.451981    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:58.526067 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.526012    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd57aa0-260a-4700-a598-da0140f60548-kube-api-access-l8cqh" pod "4fd57aa0-260a-4700-a598-da0140f60548" (UID: "4fd57aa0-260a-4700-a598-da0140f60548"). InnerVolumeSpecName "kube-api-access-l8cqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:08:58.575452 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.542267    5070 generic.go:356] "Generic (PLEG): container finished" podID="f832372d-9cd8-466b-a057-5f06e3e372ea" containerID="0024023d85a82fa0aec464eb3c50ae65ae41a01be8a38dcd0bb643a2e157dcff" exitCode=0
Nov 03 08:08:58.576787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.542633    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf" event={"ID":"f832372d-9cd8-466b-a057-5f06e3e372ea","Type":"ContainerDied","Data":"0024023d85a82fa0aec464eb3c50ae65ae41a01be8a38dcd0bb643a2e157dcff"}
Nov 03 08:08:58.581228 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.579961    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:58.590814 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.590237    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8cqh\" (UniqueName: \"kubernetes.io/projected/4fd57aa0-260a-4700-a598-da0140f60548-kube-api-access-l8cqh\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:08:58.600976 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.600941    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:08:58.609703 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.609222    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx" event={"ID":"4fd57aa0-260a-4700-a598-da0140f60548","Type":"ContainerDied","Data":"08c1a26481f6a82d34e11f8f1b44bdc8a1dc8473160a6de4523e1588bd9fabff"}
Nov 03 08:08:58.609703 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.609283    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08c1a26481f6a82d34e11f8f1b44bdc8a1dc8473160a6de4523e1588bd9fabff"
Nov 03 08:08:58.609703 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.609370    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx"
Nov 03 08:08:58.620447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.620425    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:08:58.676791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.670096    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:58.826227 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.826187    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:08:58.874829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.868474    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:08:58.893781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.892620    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:08:58.904793 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.904270    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:08:58.906891 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.906860    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:08:58.923889 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.923869    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:08:58.963264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.963241    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb"]
Nov 03 08:08:58.981349 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.981327    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:08:58.993379 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:58.993353    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:08:59.058911 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:59.058869    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:08:59.267106 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:59.267077    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf"
Nov 03 08:08:59.370312 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:59.370283    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f832372d-9cd8-466b-a057-5f06e3e372ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f832372d-9cd8-466b-a057-5f06e3e372ea-maas-api-openshift-ca\") pod \"f832372d-9cd8-466b-a057-5f06e3e372ea\" (UID: \"f832372d-9cd8-466b-a057-5f06e3e372ea\") "
Nov 03 08:08:59.370515 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:59.370426    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f832372d-9cd8-466b-a057-5f06e3e372ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f832372d-9cd8-466b-a057-5f06e3e372ea-maas-api-xks-ca\") pod \"f832372d-9cd8-466b-a057-5f06e3e372ea\" (UID: \"f832372d-9cd8-466b-a057-5f06e3e372ea\") "
Nov 03 08:08:59.370515 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:59.370461    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f832372d-9cd8-466b-a057-5f06e3e372ea-kube-api-access-n84vj\" (UniqueName: \"kubernetes.io/projected/f832372d-9cd8-466b-a057-5f06e3e372ea-kube-api-access-n84vj\") pod \"f832372d-9cd8-466b-a057-5f06e3e372ea\" (UID: \"f832372d-9cd8-466b-a057-5f06e3e372ea\") "
Nov 03 08:08:59.370705 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:59.370677    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f832372d-9cd8-466b-a057-5f06e3e372ea-maas-api-openshift-ca" pod "f832372d-9cd8-466b-a057-5f06e3e372ea" (UID: "f832372d-9cd8-466b-a057-5f06e3e372ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:08:59.371184 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:59.371162    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f832372d-9cd8-466b-a057-5f06e3e372ea-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:08:59.372488 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:59.372464    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f832372d-9cd8-466b-a057-5f06e3e372ea-maas-api-xks-ca" pod "f832372d-9cd8-466b-a057-5f06e3e372ea" (UID: "f832372d-9cd8-466b-a057-5f06e3e372ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:08:59.372699 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:59.372674    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f832372d-9cd8-466b-a057-5f06e3e372ea-kube-api-access-n84vj" pod "f832372d-9cd8-466b-a057-5f06e3e372ea" (UID: "f832372d-9cd8-466b-a057-5f06e3e372ea"). InnerVolumeSpecName "kube-api-access-n84vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:08:59.472471 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:59.472440    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f832372d-9cd8-466b-a057-5f06e3e372ea-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:08:59.472471 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:59.472469    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n84vj\" (UniqueName: \"kubernetes.io/projected/f832372d-9cd8-466b-a057-5f06e3e372ea-kube-api-access-n84vj\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:08:59.631241 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:59.631207    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf"
Nov 03 08:08:59.631461 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:59.631303    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf" event={"ID":"f832372d-9cd8-466b-a057-5f06e3e372ea","Type":"ContainerDied","Data":"5e7db7f9e02b59bf5703ab3817895dfef89e7fdd52d5b2132fe304fb6c59b8b1"}
Nov 03 08:08:59.631461 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:59.631404    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e7db7f9e02b59bf5703ab3817895dfef89e7fdd52d5b2132fe304fb6c59b8b1"
Nov 03 08:08:59.633301 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:59.633279    5070 generic.go:356] "Generic (PLEG): container finished" podID="d45d5825-e024-4922-8ab1-ba2d417dd6dc" containerID="b82fb17dca49c9a22e3d4cb4d1af5726eca1040a410f06f6e30ef8f01d029ac4" exitCode=0
Nov 03 08:08:59.633481 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:08:59.633466    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb" event={"ID":"d45d5825-e024-4922-8ab1-ba2d417dd6dc","Type":"ContainerDied","Data":"b82fb17dca49c9a22e3d4cb4d1af5726eca1040a410f06f6e30ef8f01d029ac4"}
Nov 03 08:09:00.844342 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:00.844310    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb"
Nov 03 08:09:00.900925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:00.900059    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d45d5825-e024-4922-8ab1-ba2d417dd6dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d45d5825-e024-4922-8ab1-ba2d417dd6dc-maas-api-openshift-ca\") pod \"d45d5825-e024-4922-8ab1-ba2d417dd6dc\" (UID: \"d45d5825-e024-4922-8ab1-ba2d417dd6dc\") "
Nov 03 08:09:00.901893 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:00.901859    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d45d5825-e024-4922-8ab1-ba2d417dd6dc-maas-api-openshift-ca" pod "d45d5825-e024-4922-8ab1-ba2d417dd6dc" (UID: "d45d5825-e024-4922-8ab1-ba2d417dd6dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:09:00.903140 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:00.902076    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d45d5825-e024-4922-8ab1-ba2d417dd6dc-kube-api-access-2g7lq\" (UniqueName: \"kubernetes.io/projected/d45d5825-e024-4922-8ab1-ba2d417dd6dc-kube-api-access-2g7lq\") pod \"d45d5825-e024-4922-8ab1-ba2d417dd6dc\" (UID: \"d45d5825-e024-4922-8ab1-ba2d417dd6dc\") "
Nov 03 08:09:00.903140 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:00.902167    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d45d5825-e024-4922-8ab1-ba2d417dd6dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d45d5825-e024-4922-8ab1-ba2d417dd6dc-maas-api-xks-ca\") pod \"d45d5825-e024-4922-8ab1-ba2d417dd6dc\" (UID: \"d45d5825-e024-4922-8ab1-ba2d417dd6dc\") "
Nov 03 08:09:00.903871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:00.903849    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d45d5825-e024-4922-8ab1-ba2d417dd6dc-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:00.905069 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:00.905033    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45d5825-e024-4922-8ab1-ba2d417dd6dc-kube-api-access-2g7lq" pod "d45d5825-e024-4922-8ab1-ba2d417dd6dc" (UID: "d45d5825-e024-4922-8ab1-ba2d417dd6dc"). InnerVolumeSpecName "kube-api-access-2g7lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:09:00.905800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:00.905750    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d45d5825-e024-4922-8ab1-ba2d417dd6dc-maas-api-xks-ca" pod "d45d5825-e024-4922-8ab1-ba2d417dd6dc" (UID: "d45d5825-e024-4922-8ab1-ba2d417dd6dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:09:01.005945 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:01.005907    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g7lq\" (UniqueName: \"kubernetes.io/projected/d45d5825-e024-4922-8ab1-ba2d417dd6dc-kube-api-access-2g7lq\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:01.005945 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:01.005939    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d45d5825-e024-4922-8ab1-ba2d417dd6dc-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:01.656080 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:01.656045    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb"
Nov 03 08:09:01.656293 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:01.656076    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb" event={"ID":"d45d5825-e024-4922-8ab1-ba2d417dd6dc","Type":"ContainerDied","Data":"e2d83dee4ad9c302dcf55c6f34f620c570d8b392a0048f70093af6682cae642b"}
Nov 03 08:09:01.656293 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:01.656131    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2d83dee4ad9c302dcf55c6f34f620c570d8b392a0048f70093af6682cae642b"
Nov 03 08:09:07.070959 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:07.070885    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292"]
Nov 03 08:09:09.074905 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:09.074864    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb"]
Nov 03 08:09:12.071307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:12.071268    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292" podUID="8f2e9c62-2b7c-452c-8915-05983bc505df" containerName="payload-processing" containerID="cri-o://54f24bd56185742af4831862b81e8b2e9f1cd27035f77107c9d2e1c92b5e9d23" gracePeriod=25
Nov 03 08:09:12.387402 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:12.387342    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx"]
Nov 03 08:09:12.398713 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:12.398679    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0a9dd59a-5lxlx"]
Nov 03 08:09:12.406033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:12.406009    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292"
Nov 03 08:09:12.557348 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:12.557316    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f2e9c62-2b7c-452c-8915-05983bc505df-kube-api-access-jlqd9\" (UniqueName: \"kubernetes.io/projected/8f2e9c62-2b7c-452c-8915-05983bc505df-kube-api-access-jlqd9\") pod \"8f2e9c62-2b7c-452c-8915-05983bc505df\" (UID: \"8f2e9c62-2b7c-452c-8915-05983bc505df\") "
Nov 03 08:09:12.557548 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:12.557515    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f2e9c62-2b7c-452c-8915-05983bc505df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2e9c62-2b7c-452c-8915-05983bc505df-plugins-config-volume\") pod \"8f2e9c62-2b7c-452c-8915-05983bc505df\" (UID: \"8f2e9c62-2b7c-452c-8915-05983bc505df\") "
Nov 03 08:09:12.558004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:12.557972    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2e9c62-2b7c-452c-8915-05983bc505df-plugins-config-volume" pod "8f2e9c62-2b7c-452c-8915-05983bc505df" (UID: "8f2e9c62-2b7c-452c-8915-05983bc505df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:09:12.558309 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:12.558286    5070 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2e9c62-2b7c-452c-8915-05983bc505df-plugins-config-volume\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:12.560215 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:12.560177    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2e9c62-2b7c-452c-8915-05983bc505df-kube-api-access-jlqd9" pod "8f2e9c62-2b7c-452c-8915-05983bc505df" (UID: "8f2e9c62-2b7c-452c-8915-05983bc505df"). InnerVolumeSpecName "kube-api-access-jlqd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:09:12.659953 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:12.659873    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlqd9\" (UniqueName: \"kubernetes.io/projected/8f2e9c62-2b7c-452c-8915-05983bc505df-kube-api-access-jlqd9\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:12.871322 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:12.871267    5070 generic.go:356] "Generic (PLEG): container finished" podID="8f2e9c62-2b7c-452c-8915-05983bc505df" containerID="54f24bd56185742af4831862b81e8b2e9f1cd27035f77107c9d2e1c92b5e9d23" exitCode=0
Nov 03 08:09:12.871593 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:12.871354    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292" event={"ID":"8f2e9c62-2b7c-452c-8915-05983bc505df","Type":"ContainerDied","Data":"54f24bd56185742af4831862b81e8b2e9f1cd27035f77107c9d2e1c92b5e9d23"}
Nov 03 08:09:12.871593 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:12.871457    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292" event={"ID":"8f2e9c62-2b7c-452c-8915-05983bc505df","Type":"ContainerDied","Data":"46f3fae82718ed2244d9fd62d0ae551d08919745b05119fdc85a595c6a46b18f"}
Nov 03 08:09:12.871593 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:12.871489    5070 scope.go:122] "RemoveContainer" containerID="54f24bd56185742af4831862b81e8b2e9f1cd27035f77107c9d2e1c92b5e9d23"
Nov 03 08:09:12.871940 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:12.871856    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292"
Nov 03 08:09:12.888855 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:12.888832    5070 scope.go:122] "RemoveContainer" containerID="54f24bd56185742af4831862b81e8b2e9f1cd27035f77107c9d2e1c92b5e9d23"
Nov 03 08:09:12.889156 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:09:12.889134    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54f24bd56185742af4831862b81e8b2e9f1cd27035f77107c9d2e1c92b5e9d23\": container with ID starting with 54f24bd56185742af4831862b81e8b2e9f1cd27035f77107c9d2e1c92b5e9d23 not found: ID does not exist" containerID="54f24bd56185742af4831862b81e8b2e9f1cd27035f77107c9d2e1c92b5e9d23"
Nov 03 08:09:12.889230 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:12.889165    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54f24bd56185742af4831862b81e8b2e9f1cd27035f77107c9d2e1c92b5e9d23"} err="failed to get container status \"54f24bd56185742af4831862b81e8b2e9f1cd27035f77107c9d2e1c92b5e9d23\": rpc error: code = NotFound desc = could not find container \"54f24bd56185742af4831862b81e8b2e9f1cd27035f77107c9d2e1c92b5e9d23\": container with ID starting with 54f24bd56185742af4831862b81e8b2e9f1cd27035f77107c9d2e1c92b5e9d23 not found: ID does not exist"
Nov 03 08:09:12.901577 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:12.901546    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292"]
Nov 03 08:09:12.909361 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:12.909335    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0a9dd59a-55569799bf-8w292"]
Nov 03 08:09:13.213068 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.213034    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf"]
Nov 03 08:09:13.220235 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.220206    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c855f06a-g9bkf"]
Nov 03 08:09:13.559239 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.559161    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"]
Nov 03 08:09:13.560016 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.559990    5070 cpu_manager.go:414] "containerMap: removing container" podUID="4fd57aa0-260a-4700-a598-da0140f60548" containerName="revoke-keys"
Nov 03 08:09:13.560016 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.560015    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd57aa0-260a-4700-a598-da0140f60548" containerName="revoke-keys"
Nov 03 08:09:13.560227 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.560042    5070 cpu_manager.go:414] "containerMap: removing container" podUID="8f2e9c62-2b7c-452c-8915-05983bc505df" containerName="payload-processing"
Nov 03 08:09:13.560227 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.560052    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f2e9c62-2b7c-452c-8915-05983bc505df" containerName="payload-processing"
Nov 03 08:09:13.560227 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.560081    5070 cpu_manager.go:414] "containerMap: removing container" podUID="d45d5825-e024-4922-8ab1-ba2d417dd6dc" containerName="revoke-keys"
Nov 03 08:09:13.560227 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.560087    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d45d5825-e024-4922-8ab1-ba2d417dd6dc" containerName="revoke-keys"
Nov 03 08:09:13.560227 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.560102    5070 cpu_manager.go:414] "containerMap: removing container" podUID="f832372d-9cd8-466b-a057-5f06e3e372ea" containerName="revoke-keys"
Nov 03 08:09:13.560227 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.560110    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f832372d-9cd8-466b-a057-5f06e3e372ea" containerName="revoke-keys"
Nov 03 08:09:13.560544 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.560265    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="d45d5825-e024-4922-8ab1-ba2d417dd6dc" containerName="revoke-keys"
Nov 03 08:09:13.560544 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.560280    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="f832372d-9cd8-466b-a057-5f06e3e372ea" containerName="revoke-keys"
Nov 03 08:09:13.560544 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.560292    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd57aa0-260a-4700-a598-da0140f60548" containerName="revoke-keys"
Nov 03 08:09:13.560544 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.560300    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f2e9c62-2b7c-452c-8915-05983bc505df" containerName="payload-processing"
Nov 03 08:09:13.567284 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.567259    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.571810 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.571631    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9b241313-openshift-default-dockercfg-9548j\""
Nov 03 08:09:13.573857 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.573833    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"]
Nov 03 08:09:13.690862 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.684194    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-podinfo\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.690862 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.684437    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-credential-socket\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.690862 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.684570    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rd4v\" (UniqueName: \"kubernetes.io/projected/1f4b6783-7917-41b2-9824-e21d34fe3601-kube-api-access-9rd4v\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.690862 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.684670    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-workload-socket\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.690862 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.684818    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-envoy\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.690862 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.684987    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f4b6783-7917-41b2-9824-e21d34fe3601-istiod-ca-cert\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.690862 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.685043    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-data\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.690862 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.685127    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-token\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.690862 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.685329    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-workload-certs\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.799780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.799564    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f4b6783-7917-41b2-9824-e21d34fe3601-istiod-ca-cert\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.799780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.799618    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-data\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.799780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.799654    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-token\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.799780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.799782    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-workload-certs\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.800171 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.799881    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-podinfo\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.800171 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.799988    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-credential-socket\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.800171 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.800052    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rd4v\" (UniqueName: \"kubernetes.io/projected/1f4b6783-7917-41b2-9824-e21d34fe3601-kube-api-access-9rd4v\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.800171 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.800132    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-workload-socket\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.800385 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.800182    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-envoy\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.809823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.801240    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-workload-certs\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.809823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.801920    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f4b6783-7917-41b2-9824-e21d34fe3601-istiod-ca-cert\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.809823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.802171    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-data\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.809823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.802712    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-credential-socket\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.820777 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.814184    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-workload-socket\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.861872 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.850603    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rd4v\" (UniqueName: \"kubernetes.io/projected/1f4b6783-7917-41b2-9824-e21d34fe3601-kube-api-access-9rd4v\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.861872 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.850705    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-envoy\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.861872 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.851050    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-podinfo\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.876017 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.874872    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-token\") pod \"e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") " pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:13.884850 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:13.884822    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:14.030017 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.029982    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fd57aa0-260a-4700-a598-da0140f60548" path="/var/lib/kubelet/pods/4fd57aa0-260a-4700-a598-da0140f60548/volumes"
Nov 03 08:09:14.031106 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.030656    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f2e9c62-2b7c-452c-8915-05983bc505df" path="/var/lib/kubelet/pods/8f2e9c62-2b7c-452c-8915-05983bc505df/volumes"
Nov 03 08:09:14.031494 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.031471    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f832372d-9cd8-466b-a057-5f06e3e372ea" path="/var/lib/kubelet/pods/f832372d-9cd8-466b-a057-5f06e3e372ea/volumes"
Nov 03 08:09:14.105117 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.096992    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb" podUID="8174368c-689f-4bb0-a8a8-3a58f8916807" containerName="payload-pre-processing" containerID="cri-o://1fcbada4ad1725ad74ed8a72eefcf44f72cb6eeeb8721797539cf0c25c98404d" gracePeriod=25
Nov 03 08:09:14.203996 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:09:14.203943    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f4b6783_7917_41b2_9824_e21d34fe3601.slice/crio-1fe134f9e53a9836d560a9610cecd738bb0b66c46df401d1491871b0cecc38ff WatchSource:0}: Error finding container 1fe134f9e53a9836d560a9610cecd738bb0b66c46df401d1491871b0cecc38ff: Status 404 returned error can't find the container with id 1fe134f9e53a9836d560a9610cecd738bb0b66c46df401d1491871b0cecc38ff
Nov 03 08:09:14.206898 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.206872    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"]
Nov 03 08:09:14.213918 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.213875    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:09:14.214567 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.213967    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:09:14.214567 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.214014    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:09:14.394839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.394813    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb"
Nov 03 08:09:14.531333 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.531296    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8174368c-689f-4bb0-a8a8-3a58f8916807-kube-api-access-b5q4n\" (UniqueName: \"kubernetes.io/projected/8174368c-689f-4bb0-a8a8-3a58f8916807-kube-api-access-b5q4n\") pod \"8174368c-689f-4bb0-a8a8-3a58f8916807\" (UID: \"8174368c-689f-4bb0-a8a8-3a58f8916807\") "
Nov 03 08:09:14.531529 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.531512    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8174368c-689f-4bb0-a8a8-3a58f8916807-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8174368c-689f-4bb0-a8a8-3a58f8916807-plugins-config-volume\") pod \"8174368c-689f-4bb0-a8a8-3a58f8916807\" (UID: \"8174368c-689f-4bb0-a8a8-3a58f8916807\") "
Nov 03 08:09:14.531937 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.531912    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8174368c-689f-4bb0-a8a8-3a58f8916807-plugins-config-volume" pod "8174368c-689f-4bb0-a8a8-3a58f8916807" (UID: "8174368c-689f-4bb0-a8a8-3a58f8916807"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:09:14.532102 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.532089    5070 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8174368c-689f-4bb0-a8a8-3a58f8916807-plugins-config-volume\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:14.533420 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.533401    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8174368c-689f-4bb0-a8a8-3a58f8916807-kube-api-access-b5q4n" pod "8174368c-689f-4bb0-a8a8-3a58f8916807" (UID: "8174368c-689f-4bb0-a8a8-3a58f8916807"). InnerVolumeSpecName "kube-api-access-b5q4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:09:14.634725 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.634513    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5q4n\" (UniqueName: \"kubernetes.io/projected/8174368c-689f-4bb0-a8a8-3a58f8916807-kube-api-access-b5q4n\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:14.955836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.952499    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799" event={"ID":"1f4b6783-7917-41b2-9824-e21d34fe3601","Type":"ContainerStarted","Data":"5a7804cb344bb0df975e37916afd3693eb4b39a97735cc2aa352d0bd5581b5f4"}
Nov 03 08:09:14.955836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.952566    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799" event={"ID":"1f4b6783-7917-41b2-9824-e21d34fe3601","Type":"ContainerStarted","Data":"1fe134f9e53a9836d560a9610cecd738bb0b66c46df401d1491871b0cecc38ff"}
Nov 03 08:09:14.970777 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.957522    5070 generic.go:356] "Generic (PLEG): container finished" podID="8174368c-689f-4bb0-a8a8-3a58f8916807" containerID="1fcbada4ad1725ad74ed8a72eefcf44f72cb6eeeb8721797539cf0c25c98404d" exitCode=0
Nov 03 08:09:14.970777 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.957618    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb" event={"ID":"8174368c-689f-4bb0-a8a8-3a58f8916807","Type":"ContainerDied","Data":"1fcbada4ad1725ad74ed8a72eefcf44f72cb6eeeb8721797539cf0c25c98404d"}
Nov 03 08:09:14.970777 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.957660    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb" event={"ID":"8174368c-689f-4bb0-a8a8-3a58f8916807","Type":"ContainerDied","Data":"87834386469c89cf8a981636797a5f844df9c4af1cb5b904c2b70c208c2a097f"}
Nov 03 08:09:14.970777 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.957690    5070 scope.go:122] "RemoveContainer" containerID="1fcbada4ad1725ad74ed8a72eefcf44f72cb6eeeb8721797539cf0c25c98404d"
Nov 03 08:09:14.970777 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:14.957910    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb"
Nov 03 08:09:15.056280 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:15.056221    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799" podStartSLOduration=2.05620006 podStartE2EDuration="2.05620006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:09:15.032133236 +0000 UTC m=+953.146597169" watchObservedRunningTime="2026-11-03 08:09:15.05620006 +0000 UTC m=+953.170663992"
Nov 03 08:09:15.074858 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:15.074828    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb"]
Nov 03 08:09:15.076886 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:15.076851    5070 scope.go:122] "RemoveContainer" containerID="1fcbada4ad1725ad74ed8a72eefcf44f72cb6eeeb8721797539cf0c25c98404d"
Nov 03 08:09:15.077423 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:09:15.077395    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fcbada4ad1725ad74ed8a72eefcf44f72cb6eeeb8721797539cf0c25c98404d\": container with ID starting with 1fcbada4ad1725ad74ed8a72eefcf44f72cb6eeeb8721797539cf0c25c98404d not found: ID does not exist" containerID="1fcbada4ad1725ad74ed8a72eefcf44f72cb6eeeb8721797539cf0c25c98404d"
Nov 03 08:09:15.077577 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:15.077550    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fcbada4ad1725ad74ed8a72eefcf44f72cb6eeeb8721797539cf0c25c98404d"} err="failed to get container status \"1fcbada4ad1725ad74ed8a72eefcf44f72cb6eeeb8721797539cf0c25c98404d\": rpc error: code = NotFound desc = could not find container \"1fcbada4ad1725ad74ed8a72eefcf44f72cb6eeeb8721797539cf0c25c98404d\": container with ID starting with 1fcbada4ad1725ad74ed8a72eefcf44f72cb6eeeb8721797539cf0c25c98404d not found: ID does not exist"
Nov 03 08:09:15.099008 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:15.095508    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0b397f45-c5f7d4c9-7t6mb"]
Nov 03 08:09:15.470158 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:15.470124    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb"]
Nov 03 08:09:15.479143 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:15.479114    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-0b397f45-z49xb"]
Nov 03 08:09:15.897784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:15.895125    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:15.960780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:15.960150    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:16.036410 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:16.035809    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8174368c-689f-4bb0-a8a8-3a58f8916807" path="/var/lib/kubelet/pods/8174368c-689f-4bb0-a8a8-3a58f8916807/volumes"
Nov 03 08:09:16.036410 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:16.036327    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d45d5825-e024-4922-8ab1-ba2d417dd6dc" path="/var/lib/kubelet/pods/d45d5825-e024-4922-8ab1-ba2d417dd6dc/volumes"
Nov 03 08:09:16.036745 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:16.036686    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:16.036915 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:16.036771    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:09:17.227848 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.225128    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"]
Nov 03 08:09:17.227848 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.226247    5070 cpu_manager.go:414] "containerMap: removing container" podUID="8174368c-689f-4bb0-a8a8-3a58f8916807" containerName="payload-pre-processing"
Nov 03 08:09:17.227848 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.226269    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8174368c-689f-4bb0-a8a8-3a58f8916807" containerName="payload-pre-processing"
Nov 03 08:09:17.227848 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.226574    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="8174368c-689f-4bb0-a8a8-3a58f8916807" containerName="payload-pre-processing"
Nov 03 08:09:17.238711 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.236008    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.239468 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.239444    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-a60004-openshift-default-dockercfg-c7nch\""
Nov 03 08:09:17.253553 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.249478    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"]
Nov 03 08:09:17.322701 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.322671    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-workload-certs\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.322855 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.322726    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-credential-socket\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.322899 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.322862    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08fbd019-0105-4baf-8c74-ae606e368876-istio-token\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.322936 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.322901    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08fbd019-0105-4baf-8c74-ae606e368876-istiod-ca-cert\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.322936 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.322928    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08fbd019-0105-4baf-8c74-ae606e368876-istio-podinfo\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.323063 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.323038    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-istio-data\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.323098 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.323085    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-istio-envoy\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.323165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.323144    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-workload-socket\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.323279 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.323224    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6s9x\" (UniqueName: \"kubernetes.io/projected/08fbd019-0105-4baf-8c74-ae606e368876-kube-api-access-j6s9x\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.455786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.447181    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08fbd019-0105-4baf-8c74-ae606e368876-istio-token\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.455786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.447281    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08fbd019-0105-4baf-8c74-ae606e368876-istiod-ca-cert\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.455786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.447347    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08fbd019-0105-4baf-8c74-ae606e368876-istio-podinfo\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.455786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.447418    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-istio-data\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.455786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.447453    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-istio-envoy\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.455786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.447525    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-workload-socket\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.455786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.447570    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6s9x\" (UniqueName: \"kubernetes.io/projected/08fbd019-0105-4baf-8c74-ae606e368876-kube-api-access-j6s9x\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.455786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.447656    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-workload-certs\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.455786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.447739    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-credential-socket\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.467784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.462170    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-istio-data\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.467784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.463296    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08fbd019-0105-4baf-8c74-ae606e368876-istiod-ca-cert\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.480280 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.471191    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-workload-socket\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.480280 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.479519    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-workload-certs\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.480280 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.479723    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-credential-socket\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.515780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.508562    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08fbd019-0105-4baf-8c74-ae606e368876-istio-token\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.521167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.521138    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-istio-envoy\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.531789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.527696    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08fbd019-0105-4baf-8c74-ae606e368876-istio-podinfo\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.561787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.555480    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6s9x\" (UniqueName: \"kubernetes.io/projected/08fbd019-0105-4baf-8c74-ae606e368876-kube-api-access-j6s9x\") pod \"e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") " pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.582800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.565047    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:17.982637 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:09:17.982605    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08fbd019_0105_4baf_8c74_ae606e368876.slice/crio-ae494f6c2d76665842de3920d4311cb147bf5250f6d7030f251cf051fc63524b WatchSource:0}: Error finding container ae494f6c2d76665842de3920d4311cb147bf5250f6d7030f251cf051fc63524b: Status 404 returned error can't find the container with id ae494f6c2d76665842de3920d4311cb147bf5250f6d7030f251cf051fc63524b
Nov 03 08:09:17.982748 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.982704    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"]
Nov 03 08:09:17.985548 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.985513    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:09:17.985654 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.985576    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:09:17.985654 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:17.985616    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:09:18.047637 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:18.047612    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx" event={"ID":"08fbd019-0105-4baf-8c74-ae606e368876","Type":"ContainerStarted","Data":"ae494f6c2d76665842de3920d4311cb147bf5250f6d7030f251cf051fc63524b"}
Nov 03 08:09:19.060473 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:19.060433    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx" event={"ID":"08fbd019-0105-4baf-8c74-ae606e368876","Type":"ContainerStarted","Data":"ed709d0dfc6afb094a5c7e3f852e8991cd96e0de1ae4f9ba5f1eca9f0d764ba9"}
Nov 03 08:09:19.079963 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:19.079920    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx" podStartSLOduration=2.079905716 podStartE2EDuration="2.079905716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:09:19.077155606 +0000 UTC m=+957.191619537" watchObservedRunningTime="2026-11-03 08:09:19.079905716 +0000 UTC m=+957.194369647"
Nov 03 08:09:19.578372 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:19.578336    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:19.583310 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:19.583277    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:20.081315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:20.081277    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:20.082351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:20.082323    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:09:22.055650 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:09:22.055618    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:09:22.066693 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:22.066665    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h"]
Nov 03 08:09:22.074035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:22.074010    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4df7568-h4l6h"]
Nov 03 08:09:24.028307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:24.028274    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9966eba6-1242-4af5-9fc3-3a906ecac70e" path="/var/lib/kubelet/pods/9966eba6-1242-4af5-9fc3-3a906ecac70e/volumes"
Nov 03 08:09:31.164410 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:31.163848    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm"]
Nov 03 08:09:31.174915 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:31.174870    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-87snj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:09:31.175910 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:31.175878    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-87snj" podUID="0c9ad9ff-3d0f-4994-829c-81fe102cc3de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:09:31.208780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:31.187874    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-tlbnz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:09:31.208780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:31.187951    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz" podUID="337a39e3-b285-4d8c-aa1e-210a6e46763a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:09:31.263802 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:31.263740    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc391d91-cjjjm"]
Nov 03 08:09:31.488839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:31.479909    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:09:31.797269 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:31.797163    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:09:31.833883 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:31.833834    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:09:32.041285 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:32.037293    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651375f7-9c24-485b-923b-1c451626cb8b" path="/var/lib/kubelet/pods/651375f7-9c24-485b-923b-1c451626cb8b/volumes"
Nov 03 08:09:39.019842 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:39.019811    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw"]
Nov 03 08:09:39.024475 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:39.024453    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw"
Nov 03 08:09:39.027468 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:39.027443    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:09:39.027657 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:39.027546    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:09:39.027657 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:39.027630    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:09:39.027888 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:39.027725    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:09:39.035163 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:39.034702    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw"]
Nov 03 08:09:39.126635 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:39.126598    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d2b392-c496-4b8e-8158-51f3fca3f6b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw\" (UID: \"85d2b392-c496-4b8e-8158-51f3fca3f6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw"
Nov 03 08:09:39.126859 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:39.126837    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bcch\" (UniqueName: \"kubernetes.io/projected/85d2b392-c496-4b8e-8158-51f3fca3f6b1-kube-api-access-4bcch\") pod \"maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw\" (UID: \"85d2b392-c496-4b8e-8158-51f3fca3f6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw"
Nov 03 08:09:39.127139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:39.127123    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d2b392-c496-4b8e-8158-51f3fca3f6b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw\" (UID: \"85d2b392-c496-4b8e-8158-51f3fca3f6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw"
Nov 03 08:09:39.228370 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:39.228346    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d2b392-c496-4b8e-8158-51f3fca3f6b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw\" (UID: \"85d2b392-c496-4b8e-8158-51f3fca3f6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw"
Nov 03 08:09:39.228489 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:39.228389    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d2b392-c496-4b8e-8158-51f3fca3f6b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw\" (UID: \"85d2b392-c496-4b8e-8158-51f3fca3f6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw"
Nov 03 08:09:39.228489 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:39.228476    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bcch\" (UniqueName: \"kubernetes.io/projected/85d2b392-c496-4b8e-8158-51f3fca3f6b1-kube-api-access-4bcch\") pod \"maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw\" (UID: \"85d2b392-c496-4b8e-8158-51f3fca3f6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw"
Nov 03 08:09:39.229039 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:39.229020    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d2b392-c496-4b8e-8158-51f3fca3f6b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw\" (UID: \"85d2b392-c496-4b8e-8158-51f3fca3f6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw"
Nov 03 08:09:39.230768 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:39.230733    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d2b392-c496-4b8e-8158-51f3fca3f6b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw\" (UID: \"85d2b392-c496-4b8e-8158-51f3fca3f6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw"
Nov 03 08:09:39.235086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:39.235066    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bcch\" (UniqueName: \"kubernetes.io/projected/85d2b392-c496-4b8e-8158-51f3fca3f6b1-kube-api-access-4bcch\") pod \"maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw\" (UID: \"85d2b392-c496-4b8e-8158-51f3fca3f6b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw"
Nov 03 08:09:39.343579 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:39.343519    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw"
Nov 03 08:09:39.623922 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:09:39.623888    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d2b392_c496_4b8e_8158_51f3fca3f6b1.slice/crio-e2732c5e688a60e0099d240cfa18d97c2a7cc3513b5786bc1bbc0f7aacd5ed86 WatchSource:0}: Error finding container e2732c5e688a60e0099d240cfa18d97c2a7cc3513b5786bc1bbc0f7aacd5ed86: Status 404 returned error can't find the container with id e2732c5e688a60e0099d240cfa18d97c2a7cc3513b5786bc1bbc0f7aacd5ed86
Nov 03 08:09:39.626115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:39.626094    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw"]
Nov 03 08:09:40.458019 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:40.457974    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw" event={"ID":"85d2b392-c496-4b8e-8158-51f3fca3f6b1","Type":"ContainerStarted","Data":"63717ea761b062bdd9ba53c28ba1e10cd9bd50a0cfeb7b832ebf629d6d5ebbe5"}
Nov 03 08:09:40.458532 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:40.458045    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw" event={"ID":"85d2b392-c496-4b8e-8158-51f3fca3f6b1","Type":"ContainerStarted","Data":"e2732c5e688a60e0099d240cfa18d97c2a7cc3513b5786bc1bbc0f7aacd5ed86"}
Nov 03 08:09:40.480281 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:40.480233    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw" podStartSLOduration=1.4802205640000001 podStartE2EDuration="1.480220564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:09:40.473269557 +0000 UTC m=+978.587733487" watchObservedRunningTime="2026-11-03 08:09:40.480220564 +0000 UTC m=+978.594684492"
Nov 03 08:09:41.473050 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:41.473020    5070 generic.go:356] "Generic (PLEG): container finished" podID="85d2b392-c496-4b8e-8158-51f3fca3f6b1" containerID="63717ea761b062bdd9ba53c28ba1e10cd9bd50a0cfeb7b832ebf629d6d5ebbe5" exitCode=0
Nov 03 08:09:41.473553 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:41.473106    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw" event={"ID":"85d2b392-c496-4b8e-8158-51f3fca3f6b1","Type":"ContainerDied","Data":"63717ea761b062bdd9ba53c28ba1e10cd9bd50a0cfeb7b832ebf629d6d5ebbe5"}
Nov 03 08:09:42.678829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.678808    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw"
Nov 03 08:09:42.751580 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.751497    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d2b392-c496-4b8e-8158-51f3fca3f6b1-kube-api-access-4bcch\" (UniqueName: \"kubernetes.io/projected/85d2b392-c496-4b8e-8158-51f3fca3f6b1-kube-api-access-4bcch\") pod \"85d2b392-c496-4b8e-8158-51f3fca3f6b1\" (UID: \"85d2b392-c496-4b8e-8158-51f3fca3f6b1\") "
Nov 03 08:09:42.751580 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.751558    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85d2b392-c496-4b8e-8158-51f3fca3f6b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d2b392-c496-4b8e-8158-51f3fca3f6b1-maas-api-openshift-ca\") pod \"85d2b392-c496-4b8e-8158-51f3fca3f6b1\" (UID: \"85d2b392-c496-4b8e-8158-51f3fca3f6b1\") "
Nov 03 08:09:42.752009 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.751659    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85d2b392-c496-4b8e-8158-51f3fca3f6b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d2b392-c496-4b8e-8158-51f3fca3f6b1-maas-api-xks-ca\") pod \"85d2b392-c496-4b8e-8158-51f3fca3f6b1\" (UID: \"85d2b392-c496-4b8e-8158-51f3fca3f6b1\") "
Nov 03 08:09:42.753259 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.753231    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85d2b392-c496-4b8e-8158-51f3fca3f6b1-maas-api-openshift-ca" pod "85d2b392-c496-4b8e-8158-51f3fca3f6b1" (UID: "85d2b392-c496-4b8e-8158-51f3fca3f6b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:09:42.754884 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.754854    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d2b392-c496-4b8e-8158-51f3fca3f6b1-kube-api-access-4bcch" pod "85d2b392-c496-4b8e-8158-51f3fca3f6b1" (UID: "85d2b392-c496-4b8e-8158-51f3fca3f6b1"). InnerVolumeSpecName "kube-api-access-4bcch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:09:42.755023 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.755003    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85d2b392-c496-4b8e-8158-51f3fca3f6b1-maas-api-xks-ca" pod "85d2b392-c496-4b8e-8158-51f3fca3f6b1" (UID: "85d2b392-c496-4b8e-8158-51f3fca3f6b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:09:42.843415 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.843385    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b241313-9mfqc"]
Nov 03 08:09:42.844131 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.844117    5070 cpu_manager.go:414] "containerMap: removing container" podUID="85d2b392-c496-4b8e-8158-51f3fca3f6b1" containerName="revoke-keys"
Nov 03 08:09:42.844178 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.844133    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85d2b392-c496-4b8e-8158-51f3fca3f6b1" containerName="revoke-keys"
Nov 03 08:09:42.844278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.844269    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="85d2b392-c496-4b8e-8158-51f3fca3f6b1" containerName="revoke-keys"
Nov 03 08:09:42.848411 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.848389    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b241313-9mfqc"
Nov 03 08:09:42.853581 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.853550    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqxzq\" (UniqueName: \"kubernetes.io/projected/c6f8dee5-ff7e-493b-9106-8e3b5766804d-kube-api-access-zqxzq\") pod \"maas-api-revoke-keys-e2e-ait-9b241313-9mfqc\" (UID: \"c6f8dee5-ff7e-493b-9106-8e3b5766804d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b241313-9mfqc"
Nov 03 08:09:42.853721 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.853704    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f8dee5-ff7e-493b-9106-8e3b5766804d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b241313-9mfqc\" (UID: \"c6f8dee5-ff7e-493b-9106-8e3b5766804d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b241313-9mfqc"
Nov 03 08:09:42.853849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.853791    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f8dee5-ff7e-493b-9106-8e3b5766804d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b241313-9mfqc\" (UID: \"c6f8dee5-ff7e-493b-9106-8e3b5766804d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b241313-9mfqc"
Nov 03 08:09:42.853920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.853867    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bcch\" (UniqueName: \"kubernetes.io/projected/85d2b392-c496-4b8e-8158-51f3fca3f6b1-kube-api-access-4bcch\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:42.853920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.853883    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85d2b392-c496-4b8e-8158-51f3fca3f6b1-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:42.853920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.853898    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85d2b392-c496-4b8e-8158-51f3fca3f6b1-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:42.854500 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.854482    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b241313-9mfqc"]
Nov 03 08:09:42.955181 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.955151    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqxzq\" (UniqueName: \"kubernetes.io/projected/c6f8dee5-ff7e-493b-9106-8e3b5766804d-kube-api-access-zqxzq\") pod \"maas-api-revoke-keys-e2e-ait-9b241313-9mfqc\" (UID: \"c6f8dee5-ff7e-493b-9106-8e3b5766804d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b241313-9mfqc"
Nov 03 08:09:42.955363 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.955348    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f8dee5-ff7e-493b-9106-8e3b5766804d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b241313-9mfqc\" (UID: \"c6f8dee5-ff7e-493b-9106-8e3b5766804d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b241313-9mfqc"
Nov 03 08:09:42.955460 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.955441    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f8dee5-ff7e-493b-9106-8e3b5766804d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b241313-9mfqc\" (UID: \"c6f8dee5-ff7e-493b-9106-8e3b5766804d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b241313-9mfqc"
Nov 03 08:09:42.956237 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.956178    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f8dee5-ff7e-493b-9106-8e3b5766804d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b241313-9mfqc\" (UID: \"c6f8dee5-ff7e-493b-9106-8e3b5766804d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b241313-9mfqc"
Nov 03 08:09:42.957886 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.957861    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f8dee5-ff7e-493b-9106-8e3b5766804d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9b241313-9mfqc\" (UID: \"c6f8dee5-ff7e-493b-9106-8e3b5766804d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b241313-9mfqc"
Nov 03 08:09:42.961909 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:42.961888    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqxzq\" (UniqueName: \"kubernetes.io/projected/c6f8dee5-ff7e-493b-9106-8e3b5766804d-kube-api-access-zqxzq\") pod \"maas-api-revoke-keys-e2e-ait-9b241313-9mfqc\" (UID: \"c6f8dee5-ff7e-493b-9106-8e3b5766804d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b241313-9mfqc"
Nov 03 08:09:43.168332 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:43.168298    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b241313-9mfqc"
Nov 03 08:09:43.448325 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:09:43.448292    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f8dee5_ff7e_493b_9106_8e3b5766804d.slice/crio-ab4c3cfebeafe00b5947e0cadda0243c36a0b063b9739a914ffc85291017b204 WatchSource:0}: Error finding container ab4c3cfebeafe00b5947e0cadda0243c36a0b063b9739a914ffc85291017b204: Status 404 returned error can't find the container with id ab4c3cfebeafe00b5947e0cadda0243c36a0b063b9739a914ffc85291017b204
Nov 03 08:09:43.450597 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:43.450572    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b241313-9mfqc"]
Nov 03 08:09:43.493167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:43.493139    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b241313-9mfqc" event={"ID":"c6f8dee5-ff7e-493b-9106-8e3b5766804d","Type":"ContainerStarted","Data":"ab4c3cfebeafe00b5947e0cadda0243c36a0b063b9739a914ffc85291017b204"}
Nov 03 08:09:43.494525 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:43.494500    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw"
Nov 03 08:09:43.494620 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:43.494547    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw" event={"ID":"85d2b392-c496-4b8e-8158-51f3fca3f6b1","Type":"ContainerDied","Data":"e2732c5e688a60e0099d240cfa18d97c2a7cc3513b5786bc1bbc0f7aacd5ed86"}
Nov 03 08:09:43.494620 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:43.494589    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2732c5e688a60e0099d240cfa18d97c2a7cc3513b5786bc1bbc0f7aacd5ed86"
Nov 03 08:09:44.505929 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:44.505887    5070 generic.go:356] "Generic (PLEG): container finished" podID="c6f8dee5-ff7e-493b-9106-8e3b5766804d" containerID="37ed135405642f3729b4edb29218e7deb72017f6c217798026d9a7835be0154f" exitCode=0
Nov 03 08:09:44.506319 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:44.505967    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b241313-9mfqc" event={"ID":"c6f8dee5-ff7e-493b-9106-8e3b5766804d","Type":"ContainerDied","Data":"37ed135405642f3729b4edb29218e7deb72017f6c217798026d9a7835be0154f"}
Nov 03 08:09:45.026772 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.026725    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17f8a711-stxs7"]
Nov 03 08:09:45.033243 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.033212    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17f8a711-stxs7"]
Nov 03 08:09:45.038434 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.038412    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bf6d7d4-2j8lz"]
Nov 03 08:09:45.043514 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.043490    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4bf6d7d4-2j8lz"]
Nov 03 08:09:45.677179 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.677148    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"]
Nov 03 08:09:45.682313 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.682292    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.697412 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.697388    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"]
Nov 03 08:09:45.712513 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.712485    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmb7g\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.712640 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.712580    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmb7g\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.712699 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.712660    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmb7g\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.712794 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.712771    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmb7g\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.712866 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.712851    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmb7g\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.712954 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.712940    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmb7g\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.713031 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.713019    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjnn4\" (UniqueName: \"kubernetes.io/projected/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-kube-api-access-zjnn4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmb7g\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.739222 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.739199    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b241313-9mfqc"
Nov 03 08:09:45.814488 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.814403    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6f8dee5-ff7e-493b-9106-8e3b5766804d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f8dee5-ff7e-493b-9106-8e3b5766804d-maas-api-xks-ca\") pod \"c6f8dee5-ff7e-493b-9106-8e3b5766804d\" (UID: \"c6f8dee5-ff7e-493b-9106-8e3b5766804d\") "
Nov 03 08:09:45.814634 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.814610    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6f8dee5-ff7e-493b-9106-8e3b5766804d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f8dee5-ff7e-493b-9106-8e3b5766804d-maas-api-openshift-ca\") pod \"c6f8dee5-ff7e-493b-9106-8e3b5766804d\" (UID: \"c6f8dee5-ff7e-493b-9106-8e3b5766804d\") "
Nov 03 08:09:45.814732 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.814690    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f8dee5-ff7e-493b-9106-8e3b5766804d-kube-api-access-zqxzq\" (UniqueName: \"kubernetes.io/projected/c6f8dee5-ff7e-493b-9106-8e3b5766804d-kube-api-access-zqxzq\") pod \"c6f8dee5-ff7e-493b-9106-8e3b5766804d\" (UID: \"c6f8dee5-ff7e-493b-9106-8e3b5766804d\") "
Nov 03 08:09:45.814824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.814733    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:09:45.815190 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.815122    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmb7g\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.815190 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.815137    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f8dee5-ff7e-493b-9106-8e3b5766804d-maas-api-openshift-ca" pod "c6f8dee5-ff7e-493b-9106-8e3b5766804d" (UID: "c6f8dee5-ff7e-493b-9106-8e3b5766804d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:09:45.815367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.815293    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmb7g\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.815425 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.815401    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjnn4\" (UniqueName: \"kubernetes.io/projected/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-kube-api-access-zjnn4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmb7g\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.815570 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.815549    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmb7g\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.815731 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.815713    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmb7g\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.815874 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.815859    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmb7g\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.815987 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.815968    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmb7g\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.816492 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.816095    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6f8dee5-ff7e-493b-9106-8e3b5766804d-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:45.816816 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.816794    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmb7g\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.818926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.818902    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmb7g\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.820059 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.820028    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmb7g\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.821199 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.821151    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmb7g\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.821199 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.821189    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmb7g\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.821345 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.821209    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f8dee5-ff7e-493b-9106-8e3b5766804d-kube-api-access-zqxzq" pod "c6f8dee5-ff7e-493b-9106-8e3b5766804d" (UID: "c6f8dee5-ff7e-493b-9106-8e3b5766804d"). InnerVolumeSpecName "kube-api-access-zqxzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:09:45.822660 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.822635    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f8dee5-ff7e-493b-9106-8e3b5766804d-maas-api-xks-ca" pod "c6f8dee5-ff7e-493b-9106-8e3b5766804d" (UID: "c6f8dee5-ff7e-493b-9106-8e3b5766804d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:09:45.824987 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.824959    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjnn4\" (UniqueName: \"kubernetes.io/projected/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-kube-api-access-zjnn4\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmb7g\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.825181 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.825160    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zmb7g\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:45.939782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.938261    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6f8dee5-ff7e-493b-9106-8e3b5766804d-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:45.939782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:45.938303    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqxzq\" (UniqueName: \"kubernetes.io/projected/c6f8dee5-ff7e-493b-9106-8e3b5766804d-kube-api-access-zqxzq\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:09:46.011790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:46.009100    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:46.110505 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:46.087450    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18b8576e-74fd-455f-bd5a-801ffa5ea5b2" path="/var/lib/kubelet/pods/18b8576e-74fd-455f-bd5a-801ffa5ea5b2/volumes"
Nov 03 08:09:46.110505 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:46.088314    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff9f5ebb-778f-4239-9168-c5706d9d2fbe" path="/var/lib/kubelet/pods/ff9f5ebb-778f-4239-9168-c5706d9d2fbe/volumes"
Nov 03 08:09:46.110505 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:46.092810    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499797c6-wznjq"]
Nov 03 08:09:46.158103 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:46.158073    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02be7f4f-trs9t"]
Nov 03 08:09:46.169853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:46.169812    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499797c6-wznjq"]
Nov 03 08:09:46.179651 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:46.179606    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02be7f4f-trs9t"]
Nov 03 08:09:46.564881 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:46.563973    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"]
Nov 03 08:09:46.574826 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:09:46.574794    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f09e1af_345e_4489_806a_b3aa7b8cfbd8.slice/crio-4b84c50a558fe47f9e6e6da0b29069f6ffd515bce92014548b3519413af23082 WatchSource:0}: Error finding container 4b84c50a558fe47f9e6e6da0b29069f6ffd515bce92014548b3519413af23082: Status 404 returned error can't find the container with id 4b84c50a558fe47f9e6e6da0b29069f6ffd515bce92014548b3519413af23082
Nov 03 08:09:46.575434 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:46.575414    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b241313-9mfqc"
Nov 03 08:09:46.575476 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:46.575453    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b241313-9mfqc" event={"ID":"c6f8dee5-ff7e-493b-9106-8e3b5766804d","Type":"ContainerDied","Data":"ab4c3cfebeafe00b5947e0cadda0243c36a0b063b9739a914ffc85291017b204"}
Nov 03 08:09:46.575508 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:46.575500    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab4c3cfebeafe00b5947e0cadda0243c36a0b063b9739a914ffc85291017b204"
Nov 03 08:09:46.577351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:46.577319    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:09:46.577434 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:46.577382    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:09:47.876831 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:47.876597    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g" event={"ID":"9f09e1af-345e-4489-806a-b3aa7b8cfbd8","Type":"ContainerStarted","Data":"cd1dd43f1a57f2fc7b32734b70073370e497c19ac9de054664734a80bb610267"}
Nov 03 08:09:47.876831 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:47.876662    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g" event={"ID":"9f09e1af-345e-4489-806a-b3aa7b8cfbd8","Type":"ContainerStarted","Data":"4b84c50a558fe47f9e6e6da0b29069f6ffd515bce92014548b3519413af23082"}
Nov 03 08:09:47.880690 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:47.878566    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:47.920152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:47.904607    5070 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-zmb7g container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:09:47.920152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:47.904699    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g" podUID="9f09e1af-345e-4489-806a-b3aa7b8cfbd8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:09:47.990809 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:47.989434    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g" podStartSLOduration=2.989413914 podStartE2EDuration="2.989413914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:09:47.905340823 +0000 UTC m=+986.019804755" watchObservedRunningTime="2026-11-03 08:09:47.989413914 +0000 UTC m=+986.103877847"
Nov 03 08:09:48.034285 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:48.034248    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab80b3c1-db72-4b77-8ac8-f8b8d6294447" path="/var/lib/kubelet/pods/ab80b3c1-db72-4b77-8ac8-f8b8d6294447/volumes"
Nov 03 08:09:48.035259 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:48.035235    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df483417-9936-4124-969e-6d1ff66b7e06" path="/var/lib/kubelet/pods/df483417-9936-4124-969e-6d1ff66b7e06/volumes"
Nov 03 08:09:48.891907 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:48.891873    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:09:52.054175 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:09:52.054136    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:09:55.940577 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:55.940542    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b241313-9mfqc"]
Nov 03 08:09:55.946513 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:55.946486    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9b241313-9mfqc"]
Nov 03 08:09:56.028582 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:56.028541    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6f8dee5-ff7e-493b-9106-8e3b5766804d" path="/var/lib/kubelet/pods/c6f8dee5-ff7e-493b-9106-8e3b5766804d/volumes"
Nov 03 08:09:56.029033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:56.029012    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1b36786-2742x"]
Nov 03 08:09:56.031721 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:56.031698    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1b36786-2742x"]
Nov 03 08:09:56.590072 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:56.590042    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"]
Nov 03 08:09:56.590267 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:56.590226    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799" podUID="1f4b6783-7917-41b2-9824-e21d34fe3601" containerName="istio-proxy" containerID="cri-o://5a7804cb344bb0df975e37916afd3693eb4b39a97735cc2aa352d0bd5581b5f4" gracePeriod=30
Nov 03 08:09:57.048780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:57.047501    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe0ba50-t52nr"]
Nov 03 08:09:57.133870 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:57.133828    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-393ef7cc-6qkps"]
Nov 03 08:09:57.152862 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:57.152828    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe0ba50-t52nr"]
Nov 03 08:09:57.200796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:57.200749    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp"]
Nov 03 08:09:57.209165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:57.209137    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-393ef7cc-6qkps"]
Nov 03 08:09:57.265746 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:57.265708    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b28de902-fg2jp"]
Nov 03 08:09:57.324894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:57.324802    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw"]
Nov 03 08:09:57.393331 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:57.392170    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c53cba8d-bc8nw"]
Nov 03 08:09:58.307336 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:58.307288    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d21bbe7-804b-4e67-93fc-94e90ce47a9f" path="/var/lib/kubelet/pods/5d21bbe7-804b-4e67-93fc-94e90ce47a9f/volumes"
Nov 03 08:09:58.308571 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:58.308534    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6104730c-9250-4000-a32b-a13dd6128ce0" path="/var/lib/kubelet/pods/6104730c-9250-4000-a32b-a13dd6128ce0/volumes"
Nov 03 08:09:58.309426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:58.309381    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69a19836-dd98-4498-bc17-86d681f1b39e" path="/var/lib/kubelet/pods/69a19836-dd98-4498-bc17-86d681f1b39e/volumes"
Nov 03 08:09:58.310753 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:58.310731    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85d2b392-c496-4b8e-8158-51f3fca3f6b1" path="/var/lib/kubelet/pods/85d2b392-c496-4b8e-8158-51f3fca3f6b1/volumes"
Nov 03 08:09:58.311565 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:09:58.311545    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ac3a802-ff04-4427-9be4-d517a61495fc" path="/var/lib/kubelet/pods/8ac3a802-ff04-4427-9be4-d517a61495fc/volumes"
Nov 03 08:10:00.031742 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:00.031707    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq"]
Nov 03 08:10:00.036813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:00.036786    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a71a5e70-45xsq"]
Nov 03 08:10:02.030808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:02.030747    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85feb58c-6d48-4e9f-96e4-0273fc44a9eb" path="/var/lib/kubelet/pods/85feb58c-6d48-4e9f-96e4-0273fc44a9eb/volumes"
Nov 03 08:10:02.066945 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:02.066904    5070 patch_prober.go:29] interesting pod/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:10:02.067113 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:02.066992    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799" podUID="1f4b6783-7917-41b2-9824-e21d34fe3601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:06.053750 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.053716    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5"]
Nov 03 08:10:06.054889 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.054861    5070 cpu_manager.go:414] "containerMap: removing container" podUID="c6f8dee5-ff7e-493b-9106-8e3b5766804d" containerName="revoke-keys"
Nov 03 08:10:06.054889 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.054886    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f8dee5-ff7e-493b-9106-8e3b5766804d" containerName="revoke-keys"
Nov 03 08:10:06.055164 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.055148    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f8dee5-ff7e-493b-9106-8e3b5766804d" containerName="revoke-keys"
Nov 03 08:10:06.060129 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.060104    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5"
Nov 03 08:10:06.062697 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.062492    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:10:06.062929 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.062845    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:10:06.066042 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.066021    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5"]
Nov 03 08:10:06.080372 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.080347    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:10:06.080480 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.080393    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:10:06.108931 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.108882    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:10:06.124407 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.124384    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb5hh\" (UniqueName: \"kubernetes.io/projected/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78-kube-api-access-cb5hh\") pod \"maas-api-revoke-sub-c0531169-lx7z5\" (UID: \"8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5"
Nov 03 08:10:06.124600 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.124574    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0531169-lx7z5\" (UID: \"8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5"
Nov 03 08:10:06.125181 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.125103    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0531169-lx7z5\" (UID: \"8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5"
Nov 03 08:10:06.226493 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.226466    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f4b6783-7917-41b2-9824-e21d34fe3601-kube-api-access-9rd4v\" (UniqueName: \"kubernetes.io/projected/1f4b6783-7917-41b2-9824-e21d34fe3601-kube-api-access-9rd4v\") pod \"1f4b6783-7917-41b2-9824-e21d34fe3601\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") "
Nov 03 08:10:06.226692 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.226604    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-podinfo\") pod \"1f4b6783-7917-41b2-9824-e21d34fe3601\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") "
Nov 03 08:10:06.226692 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.226688    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-credential-socket\") pod \"1f4b6783-7917-41b2-9824-e21d34fe3601\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") "
Nov 03 08:10:06.226902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.226796    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-envoy\") pod \"1f4b6783-7917-41b2-9824-e21d34fe3601\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") "
Nov 03 08:10:06.226902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.226852    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-token\" (UniqueName: \"kubernetes.io/projected/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-token\") pod \"1f4b6783-7917-41b2-9824-e21d34fe3601\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") "
Nov 03 08:10:06.226902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.226895    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-workload-certs\") pod \"1f4b6783-7917-41b2-9824-e21d34fe3601\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") "
Nov 03 08:10:06.227069 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.226977    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-workload-socket\") pod \"1f4b6783-7917-41b2-9824-e21d34fe3601\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") "
Nov 03 08:10:06.227157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.227138    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f4b6783-7917-41b2-9824-e21d34fe3601-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f4b6783-7917-41b2-9824-e21d34fe3601-istiod-ca-cert\") pod \"1f4b6783-7917-41b2-9824-e21d34fe3601\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") "
Nov 03 08:10:06.227229 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.227152    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-credential-socket" pod "1f4b6783-7917-41b2-9824-e21d34fe3601" (UID: "1f4b6783-7917-41b2-9824-e21d34fe3601"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:10:06.227229 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.227184    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-data\") pod \"1f4b6783-7917-41b2-9824-e21d34fe3601\" (UID: \"1f4b6783-7917-41b2-9824-e21d34fe3601\") "
Nov 03 08:10:06.227517 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.227477    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0531169-lx7z5\" (UID: \"8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5"
Nov 03 08:10:06.227682 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.227523    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb5hh\" (UniqueName: \"kubernetes.io/projected/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78-kube-api-access-cb5hh\") pod \"maas-api-revoke-sub-c0531169-lx7z5\" (UID: \"8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5"
Nov 03 08:10:06.227682 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.227651    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0531169-lx7z5\" (UID: \"8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5"
Nov 03 08:10:06.227962 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.227940    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-workload-certs" pod "1f4b6783-7917-41b2-9824-e21d34fe3601" (UID: "1f4b6783-7917-41b2-9824-e21d34fe3601"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:10:06.228103 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.228081    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:06.228318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.228297    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-data" pod "1f4b6783-7917-41b2-9824-e21d34fe3601" (UID: "1f4b6783-7917-41b2-9824-e21d34fe3601"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:10:06.228616 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.228597    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-workload-socket" pod "1f4b6783-7917-41b2-9824-e21d34fe3601" (UID: "1f4b6783-7917-41b2-9824-e21d34fe3601"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:10:06.229098 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.229075    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f4b6783-7917-41b2-9824-e21d34fe3601-istiod-ca-cert" pod "1f4b6783-7917-41b2-9824-e21d34fe3601" (UID: "1f4b6783-7917-41b2-9824-e21d34fe3601"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:10:06.229418 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.229392    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-podinfo" pod "1f4b6783-7917-41b2-9824-e21d34fe3601" (UID: "1f4b6783-7917-41b2-9824-e21d34fe3601"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:10:06.229505 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.229466    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0531169-lx7z5\" (UID: \"8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5"
Nov 03 08:10:06.230120 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.230097    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-envoy" pod "1f4b6783-7917-41b2-9824-e21d34fe3601" (UID: "1f4b6783-7917-41b2-9824-e21d34fe3601"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:10:06.230206 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.230119    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4b6783-7917-41b2-9824-e21d34fe3601-kube-api-access-9rd4v" pod "1f4b6783-7917-41b2-9824-e21d34fe3601" (UID: "1f4b6783-7917-41b2-9824-e21d34fe3601"). InnerVolumeSpecName "kube-api-access-9rd4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:10:06.230206 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.230100    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-token" pod "1f4b6783-7917-41b2-9824-e21d34fe3601" (UID: "1f4b6783-7917-41b2-9824-e21d34fe3601"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:10:06.231188 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.231167    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0531169-lx7z5\" (UID: \"8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5"
Nov 03 08:10:06.235070 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.235052    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb5hh\" (UniqueName: \"kubernetes.io/projected/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78-kube-api-access-cb5hh\") pod \"maas-api-revoke-sub-c0531169-lx7z5\" (UID: \"8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5"
Nov 03 08:10:06.330874 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.330834    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:06.330874 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.330875    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:06.331157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.330889    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:06.331157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.330903    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:06.331157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.330918    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f4b6783-7917-41b2-9824-e21d34fe3601-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:06.331157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.330930    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:06.331157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.330944    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rd4v\" (UniqueName: \"kubernetes.io/projected/1f4b6783-7917-41b2-9824-e21d34fe3601-kube-api-access-9rd4v\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:06.331157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.330958    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f4b6783-7917-41b2-9824-e21d34fe3601-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:06.408028 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.407930    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5"
Nov 03 08:10:06.540504 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.540460    5070 generic.go:356] "Generic (PLEG): container finished" podID="1f4b6783-7917-41b2-9824-e21d34fe3601" containerID="5a7804cb344bb0df975e37916afd3693eb4b39a97735cc2aa352d0bd5581b5f4" exitCode=0
Nov 03 08:10:06.540682 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.540512    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"
Nov 03 08:10:06.540682 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.540545    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799" event={"ID":"1f4b6783-7917-41b2-9824-e21d34fe3601","Type":"ContainerDied","Data":"5a7804cb344bb0df975e37916afd3693eb4b39a97735cc2aa352d0bd5581b5f4"}
Nov 03 08:10:06.540682 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.540613    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799" event={"ID":"1f4b6783-7917-41b2-9824-e21d34fe3601","Type":"ContainerDied","Data":"1fe134f9e53a9836d560a9610cecd738bb0b66c46df401d1491871b0cecc38ff"}
Nov 03 08:10:06.540682 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.540647    5070 scope.go:122] "RemoveContainer" containerID="5a7804cb344bb0df975e37916afd3693eb4b39a97735cc2aa352d0bd5581b5f4"
Nov 03 08:10:06.557714 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.557690    5070 scope.go:122] "RemoveContainer" containerID="5a7804cb344bb0df975e37916afd3693eb4b39a97735cc2aa352d0bd5581b5f4"
Nov 03 08:10:06.558052 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:10:06.558021    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a7804cb344bb0df975e37916afd3693eb4b39a97735cc2aa352d0bd5581b5f4\": container with ID starting with 5a7804cb344bb0df975e37916afd3693eb4b39a97735cc2aa352d0bd5581b5f4 not found: ID does not exist" containerID="5a7804cb344bb0df975e37916afd3693eb4b39a97735cc2aa352d0bd5581b5f4"
Nov 03 08:10:06.558172 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.558060    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a7804cb344bb0df975e37916afd3693eb4b39a97735cc2aa352d0bd5581b5f4"} err="failed to get container status \"5a7804cb344bb0df975e37916afd3693eb4b39a97735cc2aa352d0bd5581b5f4\": rpc error: code = NotFound desc = could not find container \"5a7804cb344bb0df975e37916afd3693eb4b39a97735cc2aa352d0bd5581b5f4\": container with ID starting with 5a7804cb344bb0df975e37916afd3693eb4b39a97735cc2aa352d0bd5581b5f4 not found: ID does not exist"
Nov 03 08:10:06.572832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.572806    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"]
Nov 03 08:10:06.578953 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.578932    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9b241313-openshift-default-5d96bcfb95-hp799"]
Nov 03 08:10:06.695018 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:10:06.694996    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a02fc3a_0a1a_45a1_a80d_e7afbd0e8b78.slice/crio-c45060e2bb9e4a9b5cd60239e7930aa303032597690ff65673f924b8aa69ad9c WatchSource:0}: Error finding container c45060e2bb9e4a9b5cd60239e7930aa303032597690ff65673f924b8aa69ad9c: Status 404 returned error can't find the container with id c45060e2bb9e4a9b5cd60239e7930aa303032597690ff65673f924b8aa69ad9c
Nov 03 08:10:06.696747 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:06.696693    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5"]
Nov 03 08:10:07.551527 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:07.551491    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5" event={"ID":"8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78","Type":"ContainerStarted","Data":"3c2ad242be5517969b3f9c1810cd708fe136eeb3bc6b6ccc69e8af73d16d59e6"}
Nov 03 08:10:07.552038 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:07.551555    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5" event={"ID":"8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78","Type":"ContainerStarted","Data":"c45060e2bb9e4a9b5cd60239e7930aa303032597690ff65673f924b8aa69ad9c"}
Nov 03 08:10:07.569728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:07.569671    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5" podStartSLOduration=1.569654952 podStartE2EDuration="1.569654952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:07.564042736 +0000 UTC m=+1005.678506667" watchObservedRunningTime="2026-11-03 08:10:07.569654952 +0000 UTC m=+1005.684118884"
Nov 03 08:10:08.027842 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:08.027808    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f4b6783-7917-41b2-9824-e21d34fe3601" path="/var/lib/kubelet/pods/1f4b6783-7917-41b2-9824-e21d34fe3601/volumes"
Nov 03 08:10:08.564175 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:08.564140    5070 generic.go:356] "Generic (PLEG): container finished" podID="8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78" containerID="3c2ad242be5517969b3f9c1810cd708fe136eeb3bc6b6ccc69e8af73d16d59e6" exitCode=0
Nov 03 08:10:08.564582 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:08.564193    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5" event={"ID":"8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78","Type":"ContainerDied","Data":"3c2ad242be5517969b3f9c1810cd708fe136eeb3bc6b6ccc69e8af73d16d59e6"}
Nov 03 08:10:09.029018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:09.028985    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e33e4b-tm2n4"]
Nov 03 08:10:09.035425 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:09.035390    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n"]
Nov 03 08:10:09.040287 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:09.040264    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbef5506-hm49w"]
Nov 03 08:10:09.044903 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:09.044884    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4e33e4b-tm2n4"]
Nov 03 08:10:09.049450 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:09.049436    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbef5506-hm49w"]
Nov 03 08:10:09.053956 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:09.053936    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaec62f7-ghj6n"]
Nov 03 08:10:09.753946 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:09.753924    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5"
Nov 03 08:10:09.898281 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:09.898254    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78-maas-api-openshift-ca\") pod \"8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78\" (UID: \"8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78\") "
Nov 03 08:10:09.898431 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:09.898410    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78-kube-api-access-cb5hh\" (UniqueName: \"kubernetes.io/projected/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78-kube-api-access-cb5hh\") pod \"8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78\" (UID: \"8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78\") "
Nov 03 08:10:09.898487 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:09.898448    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78-maas-api-xks-ca\") pod \"8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78\" (UID: \"8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78\") "
Nov 03 08:10:09.898673 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:09.898652    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78-maas-api-openshift-ca" pod "8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78" (UID: "8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:10:09.899222 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:09.899197    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:09.900584 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:09.900541    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78-kube-api-access-cb5hh" pod "8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78" (UID: "8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78"). InnerVolumeSpecName "kube-api-access-cb5hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:10:09.900749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:09.900728    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78-maas-api-xks-ca" pod "8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78" (UID: "8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:10:10.016779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:10.011206    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb5hh\" (UniqueName: \"kubernetes.io/projected/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78-kube-api-access-cb5hh\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:10.016779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:10.011249    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:10:10.076042 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:10.076005    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fbc395a-05ae-4c54-a6a6-acdbc8e29f00" path="/var/lib/kubelet/pods/3fbc395a-05ae-4c54-a6a6-acdbc8e29f00/volumes"
Nov 03 08:10:10.076940 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:10.076913    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7db497ac-2392-4e2d-9b56-0398213ced6c" path="/var/lib/kubelet/pods/7db497ac-2392-4e2d-9b56-0398213ced6c/volumes"
Nov 03 08:10:10.079128 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:10.077720    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3" path="/var/lib/kubelet/pods/9ed4d3f4-88e6-47a0-a3b9-906b570fa3c3/volumes"
Nov 03 08:10:10.589076 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:10.589032    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5" event={"ID":"8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78","Type":"ContainerDied","Data":"c45060e2bb9e4a9b5cd60239e7930aa303032597690ff65673f924b8aa69ad9c"}
Nov 03 08:10:10.589255 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:10.589082    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5"
Nov 03 08:10:10.589255 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:10.589106    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c45060e2bb9e4a9b5cd60239e7930aa303032597690ff65673f924b8aa69ad9c"
Nov 03 08:10:11.066784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:11.043226    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5"]
Nov 03 08:10:11.086661 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:11.086619    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd7f375-spmj5"]
Nov 03 08:10:12.026808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:12.026776    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a18e750c-1f55-4bb7-8465-c5c03c406ee6" path="/var/lib/kubelet/pods/a18e750c-1f55-4bb7-8465-c5c03c406ee6/volumes"
Nov 03 08:10:13.411590 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.411548    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"]
Nov 03 08:10:13.412736 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.412707    5070 cpu_manager.go:414] "containerMap: removing container" podUID="1f4b6783-7917-41b2-9824-e21d34fe3601" containerName="istio-proxy"
Nov 03 08:10:13.412736 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.412735    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f4b6783-7917-41b2-9824-e21d34fe3601" containerName="istio-proxy"
Nov 03 08:10:13.413046 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.412800    5070 cpu_manager.go:414] "containerMap: removing container" podUID="8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78" containerName="revoke-keys"
Nov 03 08:10:13.413046 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.412810    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78" containerName="revoke-keys"
Nov 03 08:10:13.413046 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.413020    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f4b6783-7917-41b2-9824-e21d34fe3601" containerName="istio-proxy"
Nov 03 08:10:13.413046 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.413039    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78" containerName="revoke-keys"
Nov 03 08:10:13.475593 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.475565    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"]
Nov 03 08:10:13.475879 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.475864    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.479629 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.479599    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cb887217-openshift-default-dockercfg-6fvp6\""
Nov 03 08:10:13.513211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.513172    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb58eee-a968-448c-9204-795ea3d444b7-istiod-ca-cert\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.513343 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.513290    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-istio-envoy\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.513407 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.513354    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-workload-certs\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.513407 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.513380    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnn5n\" (UniqueName: \"kubernetes.io/projected/5bb58eee-a968-448c-9204-795ea3d444b7-kube-api-access-lnn5n\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.513518 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.513476    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb58eee-a968-448c-9204-795ea3d444b7-istio-token\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.513606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.513586    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-credential-socket\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.513829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.513806    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-workload-socket\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.513932 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.513915    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb58eee-a968-448c-9204-795ea3d444b7-istio-podinfo\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.513997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.513949    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-istio-data\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.624631 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.623950    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb58eee-a968-448c-9204-795ea3d444b7-istio-token\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.624631 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.624080    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-credential-socket\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.624631 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.624114    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-workload-socket\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.624631 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.624218    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb58eee-a968-448c-9204-795ea3d444b7-istio-podinfo\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.624631 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.624251    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-istio-data\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.624631 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.624311    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb58eee-a968-448c-9204-795ea3d444b7-istiod-ca-cert\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.624631 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.624407    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-istio-envoy\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.624631 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.624516    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-workload-certs\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.624631 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.624542    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnn5n\" (UniqueName: \"kubernetes.io/projected/5bb58eee-a968-448c-9204-795ea3d444b7-kube-api-access-lnn5n\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.643658 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.626298    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-credential-socket\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.643658 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.626913    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-workload-socket\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.646744 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.646716    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-istio-data\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.647603 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.647581    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb58eee-a968-448c-9204-795ea3d444b7-istiod-ca-cert\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.662010 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.651671    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-workload-certs\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.662010 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.661855    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb58eee-a968-448c-9204-795ea3d444b7-istio-podinfo\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.744321 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.743862    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnn5n\" (UniqueName: \"kubernetes.io/projected/5bb58eee-a968-448c-9204-795ea3d444b7-kube-api-access-lnn5n\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.744523 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.744498    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb58eee-a968-448c-9204-795ea3d444b7-istio-token\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.744984 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.744949    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-istio-envoy\") pod \"e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") " pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:13.800129 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:13.800082    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:14.124933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:14.124292    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"]
Nov 03 08:10:14.131744 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:14.131700    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:10:14.132014 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:14.131986    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:10:14.132207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:14.132183    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:10:14.759214 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:14.759175    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8" event={"ID":"5bb58eee-a968-448c-9204-795ea3d444b7","Type":"ContainerStarted","Data":"d48a352dce138a44dd9449506518e58e8571274844b5146526244f72ff5dbfe9"}
Nov 03 08:10:14.759908 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:14.759864    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8" event={"ID":"5bb58eee-a968-448c-9204-795ea3d444b7","Type":"ContainerStarted","Data":"e9ea20b8ac6b364f2eedb9cf9553a6b998c3be610c4340c63070c4badef159a6"}
Nov 03 08:10:14.795780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:14.793335    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8" podStartSLOduration=1.793318797 podStartE2EDuration="1.793318797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:14.781198375 +0000 UTC m=+1012.895662307" watchObservedRunningTime="2026-11-03 08:10:14.793318797 +0000 UTC m=+1012.907782729"
Nov 03 08:10:14.803874 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:14.803841    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:15.806949 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:15.806921    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:16.779009 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:16.778980    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:16.779893 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:16.779873    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:10:22.034826 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:22.034791    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx"]
Nov 03 08:10:22.041669 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:22.041633    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75f83b59-l54bx"]
Nov 03 08:10:22.060934 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:10:22.060900    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:10:22.253580 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:22.253546    5070 scope.go:122] "RemoveContainer" containerID="e8da2e9d914e725f2254749300c54fbcc682df4a346d5646465518b8b184879f"
Nov 03 08:10:24.030731 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:24.030698    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4397fae5-589b-4a11-9bfc-fefce07a5ab6" path="/var/lib/kubelet/pods/4397fae5-589b-4a11-9bfc-fefce07a5ab6/volumes"
Nov 03 08:10:24.031228 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:24.031212    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l"]
Nov 03 08:10:24.039712 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:24.039688    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95"]
Nov 03 08:10:24.044446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:24.044429    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51797970-6m56l"]
Nov 03 08:10:24.049110 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:24.049089    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e65e7fd-jmq95"]
Nov 03 08:10:26.030876 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:26.030832    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d" path="/var/lib/kubelet/pods/106f4ae3-6ac4-4a3c-bdc7-2000e1fc7e9d/volumes"
Nov 03 08:10:26.031589 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:26.031570    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fd1cadf-e4be-4d9c-be54-b67c399b0556" path="/var/lib/kubelet/pods/6fd1cadf-e4be-4d9c-be54-b67c399b0556/volumes"
Nov 03 08:10:27.521182 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.520808    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"]
Nov 03 08:10:27.536989 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.536959    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.544115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.544088    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"]
Nov 03 08:10:27.544386 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.544366    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-e81ace-openshift-default-dockercfg-r7xb5\""
Nov 03 08:10:27.576136 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.576112    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-workload-socket\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.576255 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.576176    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157b7ff4-503c-4ecd-908c-2c078e875322-istio-token\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.576300 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.576276    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-credential-socket\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.576416 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.576397    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2tp6\" (UniqueName: \"kubernetes.io/projected/157b7ff4-503c-4ecd-908c-2c078e875322-kube-api-access-n2tp6\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.576542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.576525    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-istio-envoy\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.576577 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.576558    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157b7ff4-503c-4ecd-908c-2c078e875322-istiod-ca-cert\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.576615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.576589    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-istio-data\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.576808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.576793    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-workload-certs\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.576869 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.576829    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157b7ff4-503c-4ecd-908c-2c078e875322-istio-podinfo\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.680007 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.679970    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-workload-certs\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.696390 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.681457    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-workload-certs\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.697259 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.697232    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157b7ff4-503c-4ecd-908c-2c078e875322-istio-podinfo\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.697999 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.697949    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-workload-socket\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.721087 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.721055    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-workload-socket\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.721388 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.721367    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157b7ff4-503c-4ecd-908c-2c078e875322-istio-token\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.721903 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.721877    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-credential-socket\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.722537 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.722515    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2tp6\" (UniqueName: \"kubernetes.io/projected/157b7ff4-503c-4ecd-908c-2c078e875322-kube-api-access-n2tp6\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.723081 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.723060    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-istio-envoy\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.723579 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.723560    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157b7ff4-503c-4ecd-908c-2c078e875322-istiod-ca-cert\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.738861 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.738828    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-istio-data\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.739383 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.724351    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157b7ff4-503c-4ecd-908c-2c078e875322-istiod-ca-cert\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.749796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.739860    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-istio-data\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.749796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.722365    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-credential-socket\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.749796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.741770    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-istio-envoy\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.757835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.750740    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157b7ff4-503c-4ecd-908c-2c078e875322-istio-podinfo\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.786049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.785963    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2tp6\" (UniqueName: \"kubernetes.io/projected/157b7ff4-503c-4ecd-908c-2c078e875322-kube-api-access-n2tp6\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.792386 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.791442    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157b7ff4-503c-4ecd-908c-2c078e875322-istio-token\") pod \"e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") " pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:27.880785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:27.880046    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:28.309593 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:28.309532    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:10:28.309768 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:28.309708    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:10:28.309847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:28.309818    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:10:28.315103 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:28.315072    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"]
Nov 03 08:10:29.139235 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:29.139198    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f" event={"ID":"157b7ff4-503c-4ecd-908c-2c078e875322","Type":"ContainerStarted","Data":"355dd9ad7c20d3e2267799e06e05f65b9b59d7b0d068944ac3522476d43805a9"}
Nov 03 08:10:29.139659 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:29.139260    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f" event={"ID":"157b7ff4-503c-4ecd-908c-2c078e875322","Type":"ContainerStarted","Data":"3274b40ffdfa693d42bf9784291f4e0532ea21f40fcecffb9ad28f309ea9def0"}
Nov 03 08:10:29.162044 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:29.161975    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f" podStartSLOduration=2.161960791 podStartE2EDuration="2.161960791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:10:29.156541297 +0000 UTC m=+1027.271005227" watchObservedRunningTime="2026-11-03 08:10:29.161960791 +0000 UTC m=+1027.276424722"
Nov 03 08:10:29.894793 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:29.887290    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:30.127783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:30.126645    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:30.182883 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:30.182194    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:30.186995 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:30.186969    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:10:52.055031 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:10:52.054947    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:10:52.984345 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:52.984310    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:10:57.253904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:57.253854    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:57.795782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:57.794925    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:57.840878 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:57.840831    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:57.842231 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:57.842198    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:58.234279 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:58.226019    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:58.277536 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:58.259329    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:58.277536 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:58.260858    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:10:58.312609 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:58.310004    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:58.333670 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:58.333626    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:58.477264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:58.477215    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:58.524213 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:58.506948    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:10:58.806903 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:58.806798    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:58.817591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:58.817561    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:10:58.842974 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:58.842913    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:58.845420 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:58.845129    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:59.169942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:59.169895    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:59.249212 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:59.249173    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:10:59.266200 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:59.266168    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:10:59.324791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:59.324570    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:59.329148 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:59.326410    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:10:59.344269 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:59.332893    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:10:59.350483 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:59.334829    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:10:59.350483 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:59.345326    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:10:59.449940 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:59.449859    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:10:59.498813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:59.455299    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:10:59.549120 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:59.549081    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:10:59.622191 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:59.622162    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:10:59.627258 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:59.627230    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:10:59.639785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:59.637977    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:10:59.652860 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:59.652836    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:10:59.725492 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:59.725415    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:10:59.749736 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:10:59.749247    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:11:02.830929 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:02.830894    5070 patch_prober.go:29] interesting pod/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:11:02.831326 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:02.830951    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8" podUID="5bb58eee-a968-448c-9204-795ea3d444b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:11:09.363288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:09.363252    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j"]
Nov 03 08:11:09.378789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:09.378751    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j"]
Nov 03 08:11:09.379005 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:09.378877    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j"
Nov 03 08:11:09.388399 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:09.384366    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:11:09.388399 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:09.384404    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:11:09.388399 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:09.384444    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:11:09.388399 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:09.384531    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:11:09.520339 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:09.520308    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b28097-fb26-499a-a6da-3dac05bee77a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a04f1d10-k2h4j\" (UID: \"11b28097-fb26-499a-a6da-3dac05bee77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j"
Nov 03 08:11:09.520518 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:09.520356    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b28097-fb26-499a-a6da-3dac05bee77a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a04f1d10-k2h4j\" (UID: \"11b28097-fb26-499a-a6da-3dac05bee77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j"
Nov 03 08:11:09.520518 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:09.520376    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6nxx\" (UniqueName: \"kubernetes.io/projected/11b28097-fb26-499a-a6da-3dac05bee77a-kube-api-access-j6nxx\") pod \"maas-api-revoke-sub-a04f1d10-k2h4j\" (UID: \"11b28097-fb26-499a-a6da-3dac05bee77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j"
Nov 03 08:11:09.622273 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:09.622203    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b28097-fb26-499a-a6da-3dac05bee77a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a04f1d10-k2h4j\" (UID: \"11b28097-fb26-499a-a6da-3dac05bee77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j"
Nov 03 08:11:09.622273 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:09.622251    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b28097-fb26-499a-a6da-3dac05bee77a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a04f1d10-k2h4j\" (UID: \"11b28097-fb26-499a-a6da-3dac05bee77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j"
Nov 03 08:11:09.622490 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:09.622274    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6nxx\" (UniqueName: \"kubernetes.io/projected/11b28097-fb26-499a-a6da-3dac05bee77a-kube-api-access-j6nxx\") pod \"maas-api-revoke-sub-a04f1d10-k2h4j\" (UID: \"11b28097-fb26-499a-a6da-3dac05bee77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j"
Nov 03 08:11:09.623078 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:09.623048    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b28097-fb26-499a-a6da-3dac05bee77a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a04f1d10-k2h4j\" (UID: \"11b28097-fb26-499a-a6da-3dac05bee77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j"
Nov 03 08:11:09.625406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:09.625382    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b28097-fb26-499a-a6da-3dac05bee77a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a04f1d10-k2h4j\" (UID: \"11b28097-fb26-499a-a6da-3dac05bee77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j"
Nov 03 08:11:09.630461 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:09.630436    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6nxx\" (UniqueName: \"kubernetes.io/projected/11b28097-fb26-499a-a6da-3dac05bee77a-kube-api-access-j6nxx\") pod \"maas-api-revoke-sub-a04f1d10-k2h4j\" (UID: \"11b28097-fb26-499a-a6da-3dac05bee77a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j"
Nov 03 08:11:09.716245 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:09.716207    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j"
Nov 03 08:11:10.094189 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:10.094160    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j"]
Nov 03 08:11:10.094912 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:11:10.094885    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b28097_fb26_499a_a6da_3dac05bee77a.slice/crio-e6fdf73321ce39d23f35984c848077c568e586fe47e43187d3c7310412bd45c4 WatchSource:0}: Error finding container e6fdf73321ce39d23f35984c848077c568e586fe47e43187d3c7310412bd45c4: Status 404 returned error can't find the container with id e6fdf73321ce39d23f35984c848077c568e586fe47e43187d3c7310412bd45c4
Nov 03 08:11:10.905065 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:10.905021    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j" event={"ID":"11b28097-fb26-499a-a6da-3dac05bee77a","Type":"ContainerStarted","Data":"3e2844dbf69aedc60417036f71a0d061759f0d3b1a3d06307336a314c37bf73e"}
Nov 03 08:11:10.905614 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:10.905115    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j" event={"ID":"11b28097-fb26-499a-a6da-3dac05bee77a","Type":"ContainerStarted","Data":"e6fdf73321ce39d23f35984c848077c568e586fe47e43187d3c7310412bd45c4"}
Nov 03 08:11:10.922459 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:10.922408    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j" podStartSLOduration=1.922395659 podStartE2EDuration="1.922395659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:11:10.918195309 +0000 UTC m=+1069.032659240" watchObservedRunningTime="2026-11-03 08:11:10.922395659 +0000 UTC m=+1069.036859590"
Nov 03 08:11:11.024146 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:11.024113    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h"]
Nov 03 08:11:11.030328 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:11.030302    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ac77225-qhd8h"]
Nov 03 08:11:11.916131 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:11.916046    5070 generic.go:356] "Generic (PLEG): container finished" podID="5bb58eee-a968-448c-9204-795ea3d444b7" containerID="d48a352dce138a44dd9449506518e58e8571274844b5146526244f72ff5dbfe9" exitCode=0
Nov 03 08:11:11.916131 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:11.916120    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8" event={"ID":"5bb58eee-a968-448c-9204-795ea3d444b7","Type":"ContainerDied","Data":"d48a352dce138a44dd9449506518e58e8571274844b5146526244f72ff5dbfe9"}
Nov 03 08:11:11.916602 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:11.916579    5070 scope.go:122] "RemoveContainer" containerID="d48a352dce138a44dd9449506518e58e8571274844b5146526244f72ff5dbfe9"
Nov 03 08:11:11.918069 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:11.918042    5070 generic.go:356] "Generic (PLEG): container finished" podID="11b28097-fb26-499a-a6da-3dac05bee77a" containerID="3e2844dbf69aedc60417036f71a0d061759f0d3b1a3d06307336a314c37bf73e" exitCode=0
Nov 03 08:11:11.918169 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:11.918112    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j" event={"ID":"11b28097-fb26-499a-a6da-3dac05bee77a","Type":"ContainerDied","Data":"3e2844dbf69aedc60417036f71a0d061759f0d3b1a3d06307336a314c37bf73e"}
Nov 03 08:11:11.918453 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:11.918429    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:11:11.918507 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:11.918494    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:11:11.918542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:11.918531    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:11:12.028423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:12.028388    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01e9078f-803f-4c08-8608-5f359bb8e3e4" path="/var/lib/kubelet/pods/01e9078f-803f-4c08-8608-5f359bb8e3e4/volumes"
Nov 03 08:11:12.930397 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:12.930358    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8" event={"ID":"5bb58eee-a968-448c-9204-795ea3d444b7","Type":"ContainerStarted","Data":"4b209588d42ea1319663a41063f7159c3050fa4cc39e16c967f02958ac542d11"}
Nov 03 08:11:13.144339 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:13.144317    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j"
Nov 03 08:11:13.293796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:13.293694    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b28097-fb26-499a-a6da-3dac05bee77a-kube-api-access-j6nxx\" (UniqueName: \"kubernetes.io/projected/11b28097-fb26-499a-a6da-3dac05bee77a-kube-api-access-j6nxx\") pod \"11b28097-fb26-499a-a6da-3dac05bee77a\" (UID: \"11b28097-fb26-499a-a6da-3dac05bee77a\") "
Nov 03 08:11:13.294033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:13.293865    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11b28097-fb26-499a-a6da-3dac05bee77a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b28097-fb26-499a-a6da-3dac05bee77a-maas-api-xks-ca\") pod \"11b28097-fb26-499a-a6da-3dac05bee77a\" (UID: \"11b28097-fb26-499a-a6da-3dac05bee77a\") "
Nov 03 08:11:13.294033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:13.293926    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11b28097-fb26-499a-a6da-3dac05bee77a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b28097-fb26-499a-a6da-3dac05bee77a-maas-api-openshift-ca\") pod \"11b28097-fb26-499a-a6da-3dac05bee77a\" (UID: \"11b28097-fb26-499a-a6da-3dac05bee77a\") "
Nov 03 08:11:13.294332 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:13.294303    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11b28097-fb26-499a-a6da-3dac05bee77a-maas-api-openshift-ca" pod "11b28097-fb26-499a-a6da-3dac05bee77a" (UID: "11b28097-fb26-499a-a6da-3dac05bee77a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:11:13.294852 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:13.294830    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b28097-fb26-499a-a6da-3dac05bee77a-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:11:13.296109 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:13.296081    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11b28097-fb26-499a-a6da-3dac05bee77a-maas-api-xks-ca" pod "11b28097-fb26-499a-a6da-3dac05bee77a" (UID: "11b28097-fb26-499a-a6da-3dac05bee77a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:11:13.296236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:13.296219    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b28097-fb26-499a-a6da-3dac05bee77a-kube-api-access-j6nxx" pod "11b28097-fb26-499a-a6da-3dac05bee77a" (UID: "11b28097-fb26-499a-a6da-3dac05bee77a"). InnerVolumeSpecName "kube-api-access-j6nxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:11:13.397980 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:13.397886    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6nxx\" (UniqueName: \"kubernetes.io/projected/11b28097-fb26-499a-a6da-3dac05bee77a-kube-api-access-j6nxx\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:11:13.397980 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:13.397932    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b28097-fb26-499a-a6da-3dac05bee77a-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:11:13.801331 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:13.801299    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:11:13.802336 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:13.802311    5070 patch_prober.go:29] interesting pod/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 08:11:13.802455 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:13.802357    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8" podUID="5bb58eee-a968-448c-9204-795ea3d444b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 08:11:13.950447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:13.950416    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j"
Nov 03 08:11:13.950881 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:13.950440    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j" event={"ID":"11b28097-fb26-499a-a6da-3dac05bee77a","Type":"ContainerDied","Data":"e6fdf73321ce39d23f35984c848077c568e586fe47e43187d3c7310412bd45c4"}
Nov 03 08:11:13.950881 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:13.950523    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6fdf73321ce39d23f35984c848077c568e586fe47e43187d3c7310412bd45c4"
Nov 03 08:11:14.801854 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:14.801817    5070 patch_prober.go:29] interesting pod/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 08:11:14.802055 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:14.801891    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8" podUID="5bb58eee-a968-448c-9204-795ea3d444b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 08:11:14.908820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:14.908780    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"]
Nov 03 08:11:14.909997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:14.909968    5070 cpu_manager.go:414] "containerMap: removing container" podUID="11b28097-fb26-499a-a6da-3dac05bee77a" containerName="revoke-keys"
Nov 03 08:11:14.909997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:14.909994    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11b28097-fb26-499a-a6da-3dac05bee77a" containerName="revoke-keys"
Nov 03 08:11:14.910290 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:14.910269    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="11b28097-fb26-499a-a6da-3dac05bee77a" containerName="revoke-keys"
Nov 03 08:11:14.915224 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:14.915203    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:14.921991 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:14.921780    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2a44ba-openshift-default-dockercfg-pwnqq\""
Nov 03 08:11:14.932821 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:14.932798    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"]
Nov 03 08:11:15.055709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.054361    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dfgb\" (UniqueName: \"kubernetes.io/projected/87e84029-163b-4605-8c14-8da36b5c6cf5-kube-api-access-9dfgb\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.055709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.054432    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-token\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.055709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.054467    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e84029-163b-4605-8c14-8da36b5c6cf5-istiod-ca-cert\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.055709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.054524    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-workload-socket\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.055709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.054576    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-credential-socket\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.055709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.054611    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-data\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.055709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.054673    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-envoy\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.055709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.054864    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-podinfo\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.055709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.054979    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-workload-certs\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.200737 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.191132    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-podinfo\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.200737 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.191259    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-workload-certs\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.200737 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.191399    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dfgb\" (UniqueName: \"kubernetes.io/projected/87e84029-163b-4605-8c14-8da36b5c6cf5-kube-api-access-9dfgb\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.200737 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.191441    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-token\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.200737 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.191472    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e84029-163b-4605-8c14-8da36b5c6cf5-istiod-ca-cert\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.200737 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.191527    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-workload-socket\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.200737 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.191580    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-credential-socket\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.200737 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.191621    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-data\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.200737 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.191682    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-envoy\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.233044 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.212315    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e84029-163b-4605-8c14-8da36b5c6cf5-istiod-ca-cert\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.236275 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.236245    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-workload-socket\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.247797 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.239276    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-data\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.247797 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.239589    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-envoy\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.247797 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.241537    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-token\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.248401 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.248345    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-workload-certs\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.248807 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.248753    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-credential-socket\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.273788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.271572    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dfgb\" (UniqueName: \"kubernetes.io/projected/87e84029-163b-4605-8c14-8da36b5c6cf5-kube-api-access-9dfgb\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.307840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.302260    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-podinfo\") pod \"e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") " pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.538136 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.538101    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:15.801230 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.801197    5070 patch_prober.go:29] interesting pod/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 08:11:15.801396 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.801253    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8" podUID="5bb58eee-a968-448c-9204-795ea3d444b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 08:11:15.992416 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:11:15.992367    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e84029_163b_4605_8c14_8da36b5c6cf5.slice/crio-a0154d76e02edf72b9a514181c68daec296cfab7353152093b76e2b98bcc915e WatchSource:0}: Error finding container a0154d76e02edf72b9a514181c68daec296cfab7353152093b76e2b98bcc915e: Status 404 returned error can't find the container with id a0154d76e02edf72b9a514181c68daec296cfab7353152093b76e2b98bcc915e
Nov 03 08:11:15.994866 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.994786    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"]
Nov 03 08:11:15.995862 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.995823    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:11:15.995965 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.995939    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:11:15.996030 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:15.995993    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:11:16.012415 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:16.012380    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng" event={"ID":"87e84029-163b-4605-8c14-8da36b5c6cf5","Type":"ContainerStarted","Data":"a0154d76e02edf72b9a514181c68daec296cfab7353152093b76e2b98bcc915e"}
Nov 03 08:11:16.495848 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:16.495810    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj"]
Nov 03 08:11:16.500119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:16.500093    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj"
Nov 03 08:11:16.503027 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:16.502638    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:11:16.503027 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:16.502650    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:11:16.503976 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:16.503956    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj"]
Nov 03 08:11:16.507832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:16.505897    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:11:16.507832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:16.506004    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:11:16.559785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:16.555865    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/158baf44-a9e1-4dba-ae8b-7a980fbd1443-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-974a1057-mz2tj\" (UID: \"158baf44-a9e1-4dba-ae8b-7a980fbd1443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj"
Nov 03 08:11:16.567903 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:16.567872    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/158baf44-a9e1-4dba-ae8b-7a980fbd1443-maas-api-xks-ca\") pod \"maas-api-revoke-sub-974a1057-mz2tj\" (UID: \"158baf44-a9e1-4dba-ae8b-7a980fbd1443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj"
Nov 03 08:11:16.579775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:16.568365    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfssv\" (UniqueName: \"kubernetes.io/projected/158baf44-a9e1-4dba-ae8b-7a980fbd1443-kube-api-access-qfssv\") pod \"maas-api-revoke-sub-974a1057-mz2tj\" (UID: \"158baf44-a9e1-4dba-ae8b-7a980fbd1443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj"
Nov 03 08:11:16.686801 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:16.686748    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfssv\" (UniqueName: \"kubernetes.io/projected/158baf44-a9e1-4dba-ae8b-7a980fbd1443-kube-api-access-qfssv\") pod \"maas-api-revoke-sub-974a1057-mz2tj\" (UID: \"158baf44-a9e1-4dba-ae8b-7a980fbd1443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj"
Nov 03 08:11:16.686964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:16.686888    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/158baf44-a9e1-4dba-ae8b-7a980fbd1443-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-974a1057-mz2tj\" (UID: \"158baf44-a9e1-4dba-ae8b-7a980fbd1443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj"
Nov 03 08:11:16.686964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:16.686951    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/158baf44-a9e1-4dba-ae8b-7a980fbd1443-maas-api-xks-ca\") pod \"maas-api-revoke-sub-974a1057-mz2tj\" (UID: \"158baf44-a9e1-4dba-ae8b-7a980fbd1443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj"
Nov 03 08:11:16.687625 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:16.687605    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/158baf44-a9e1-4dba-ae8b-7a980fbd1443-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-974a1057-mz2tj\" (UID: \"158baf44-a9e1-4dba-ae8b-7a980fbd1443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj"
Nov 03 08:11:16.689603 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:16.689578    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/158baf44-a9e1-4dba-ae8b-7a980fbd1443-maas-api-xks-ca\") pod \"maas-api-revoke-sub-974a1057-mz2tj\" (UID: \"158baf44-a9e1-4dba-ae8b-7a980fbd1443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj"
Nov 03 08:11:16.693718 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:16.693690    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfssv\" (UniqueName: \"kubernetes.io/projected/158baf44-a9e1-4dba-ae8b-7a980fbd1443-kube-api-access-qfssv\") pod \"maas-api-revoke-sub-974a1057-mz2tj\" (UID: \"158baf44-a9e1-4dba-ae8b-7a980fbd1443\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj"
Nov 03 08:11:16.784891 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:16.783942    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:11:16.801506 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:16.801480    5070 patch_prober.go:29] interesting pod/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 08:11:16.801630 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:16.801526    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8" podUID="5bb58eee-a968-448c-9204-795ea3d444b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 08:11:16.820890 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:16.820858    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj"
Nov 03 08:11:17.079833 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:17.055711    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng" event={"ID":"87e84029-163b-4605-8c14-8da36b5c6cf5","Type":"ContainerStarted","Data":"805447d0005e492d53887463bb38cc18474ffbe1a7a32d179f76767976b14712"}
Nov 03 08:11:17.100367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:17.100191    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng" podStartSLOduration=3.100177469 podStartE2EDuration="3.100177469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:11:17.096548923 +0000 UTC m=+1075.211012856" watchObservedRunningTime="2026-11-03 08:11:17.100177469 +0000 UTC m=+1075.214641396"
Nov 03 08:11:17.313528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:17.313408    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj"]
Nov 03 08:11:17.539226 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:17.539198    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:17.545910 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:17.545882    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:17.801716 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:17.801678    5070 patch_prober.go:29] interesting pod/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body=
Nov 03 08:11:17.801957 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:17.801740    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8" podUID="5bb58eee-a968-448c-9204-795ea3d444b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused"
Nov 03 08:11:18.092392 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:18.092355    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj" event={"ID":"158baf44-a9e1-4dba-ae8b-7a980fbd1443","Type":"ContainerStarted","Data":"122c7c5ef4ec7bc47b6873ca187ce4168aac0e66b1ced13cb56eb2b38d3ca4a0"}
Nov 03 08:11:18.092600 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:18.092415    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj" event={"ID":"158baf44-a9e1-4dba-ae8b-7a980fbd1443","Type":"ContainerStarted","Data":"7324875b51e059da44583ca030513866bafbf5221ca23337abb2b9e1aff912b7"}
Nov 03 08:11:18.092926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:18.092899    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:18.094204 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:18.094182    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:11:18.111859 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:18.111819    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj" podStartSLOduration=2.11180772 podStartE2EDuration="2.11180772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:11:18.104827727 +0000 UTC m=+1076.219291655" watchObservedRunningTime="2026-11-03 08:11:18.11180772 +0000 UTC m=+1076.226271658"
Nov 03 08:11:18.805838 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:18.805807    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:11:18.806887 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:18.806865    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:11:19.105567 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:19.105537    5070 generic.go:356] "Generic (PLEG): container finished" podID="158baf44-a9e1-4dba-ae8b-7a980fbd1443" containerID="122c7c5ef4ec7bc47b6873ca187ce4168aac0e66b1ced13cb56eb2b38d3ca4a0" exitCode=0
Nov 03 08:11:19.105744 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:19.105617    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj" event={"ID":"158baf44-a9e1-4dba-ae8b-7a980fbd1443","Type":"ContainerDied","Data":"122c7c5ef4ec7bc47b6873ca187ce4168aac0e66b1ced13cb56eb2b38d3ca4a0"}
Nov 03 08:11:20.054705 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:20.054667    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b742d02c-kj9hm"]
Nov 03 08:11:20.062514 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:20.062480    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b742d02c-kj9hm"]
Nov 03 08:11:20.330594 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:20.330565    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj"
Nov 03 08:11:20.449960 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:20.449925    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158baf44-a9e1-4dba-ae8b-7a980fbd1443-kube-api-access-qfssv\" (UniqueName: \"kubernetes.io/projected/158baf44-a9e1-4dba-ae8b-7a980fbd1443-kube-api-access-qfssv\") pod \"158baf44-a9e1-4dba-ae8b-7a980fbd1443\" (UID: \"158baf44-a9e1-4dba-ae8b-7a980fbd1443\") "
Nov 03 08:11:20.450152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:20.450016    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/158baf44-a9e1-4dba-ae8b-7a980fbd1443-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/158baf44-a9e1-4dba-ae8b-7a980fbd1443-maas-api-xks-ca\") pod \"158baf44-a9e1-4dba-ae8b-7a980fbd1443\" (UID: \"158baf44-a9e1-4dba-ae8b-7a980fbd1443\") "
Nov 03 08:11:20.450152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:20.450070    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/158baf44-a9e1-4dba-ae8b-7a980fbd1443-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/158baf44-a9e1-4dba-ae8b-7a980fbd1443-maas-api-openshift-ca\") pod \"158baf44-a9e1-4dba-ae8b-7a980fbd1443\" (UID: \"158baf44-a9e1-4dba-ae8b-7a980fbd1443\") "
Nov 03 08:11:20.450740 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:20.450711    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158baf44-a9e1-4dba-ae8b-7a980fbd1443-maas-api-openshift-ca" pod "158baf44-a9e1-4dba-ae8b-7a980fbd1443" (UID: "158baf44-a9e1-4dba-ae8b-7a980fbd1443"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:11:20.451098 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:20.451070    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/158baf44-a9e1-4dba-ae8b-7a980fbd1443-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:11:20.453042 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:20.453010    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/158baf44-a9e1-4dba-ae8b-7a980fbd1443-maas-api-xks-ca" pod "158baf44-a9e1-4dba-ae8b-7a980fbd1443" (UID: "158baf44-a9e1-4dba-ae8b-7a980fbd1443"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:11:20.453483 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:20.453392    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158baf44-a9e1-4dba-ae8b-7a980fbd1443-kube-api-access-qfssv" pod "158baf44-a9e1-4dba-ae8b-7a980fbd1443" (UID: "158baf44-a9e1-4dba-ae8b-7a980fbd1443"). InnerVolumeSpecName "kube-api-access-qfssv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:11:20.552536 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:20.552494    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfssv\" (UniqueName: \"kubernetes.io/projected/158baf44-a9e1-4dba-ae8b-7a980fbd1443-kube-api-access-qfssv\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:11:20.552536 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:20.552539    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/158baf44-a9e1-4dba-ae8b-7a980fbd1443-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:11:21.253778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:21.241867    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj" event={"ID":"158baf44-a9e1-4dba-ae8b-7a980fbd1443","Type":"ContainerDied","Data":"7324875b51e059da44583ca030513866bafbf5221ca23337abb2b9e1aff912b7"}
Nov 03 08:11:21.253778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:21.241944    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7324875b51e059da44583ca030513866bafbf5221ca23337abb2b9e1aff912b7"
Nov 03 08:11:21.253778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:21.242017    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj"
Nov 03 08:11:22.078789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:22.074650    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53da5571-e475-4695-9764-956e91f25f21" path="/var/lib/kubelet/pods/53da5571-e475-4695-9764-956e91f25f21/volumes"
Nov 03 08:11:22.115830 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:11:22.112370    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:11:22.520779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:22.502203    5070 scope.go:122] "RemoveContainer" containerID="e4f41305f818c2ad9cc7813569a1073171b9f50ff470a454b6af4857c1aa7b0e"
Nov 03 08:11:22.612078 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:22.612054    5070 scope.go:122] "RemoveContainer" containerID="e43182984345f5f147d9fc3dfe43c833e1b9a7db8c3f9854624b5babfa8ffc3c"
Nov 03 08:11:22.695968 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:22.695555    5070 scope.go:122] "RemoveContainer" containerID="27755c39299093d7bf79a29651e39c3c497060438cb6b14cf5c9154b62f4790c"
Nov 03 08:11:22.830558 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:22.830528    5070 scope.go:122] "RemoveContainer" containerID="3aa707899708a2ef8d335fad977ffb0ee0e467dd4aa4c991f9774a13bbaf71f3"
Nov 03 08:11:22.850953 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:22.850119    5070 scope.go:122] "RemoveContainer" containerID="e85d13146ad67eceb3dcbae6604eff49ac6be873d1c3294edca35c63c59e5723"
Nov 03 08:11:22.878035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:22.870543    5070 scope.go:122] "RemoveContainer" containerID="dec8914c335f9e8567b88277e07ba95958ae5cf7af696e9e84de45c2d4d59dd1"
Nov 03 08:11:22.904429 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:22.904400    5070 scope.go:122] "RemoveContainer" containerID="befd666de4c09a9bb15ddef0fd029234f7f2469b5980d9a013523c17043b16e5"
Nov 03 08:11:22.918566 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:22.918531    5070 scope.go:122] "RemoveContainer" containerID="0ebfec6bb88f79329e61eb5a0c009df297b40c4118285b2490134a2061052eae"
Nov 03 08:11:22.931923 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:22.931897    5070 scope.go:122] "RemoveContainer" containerID="00ba44e6f4690abe1a92b3cacf20792fe0d2b6775b6480938a3ae15f1ea43c9b"
Nov 03 08:11:22.947012 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:22.946954    5070 scope.go:122] "RemoveContainer" containerID="f763aada32150c95eb2564f029424b6fd00a7f7389b984e19358606b3b183e15"
Nov 03 08:11:22.970015 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:22.962525    5070 scope.go:122] "RemoveContainer" containerID="ded35c498d485f28149a61be7fcd1767503ce1885bc0005d40c1ab726179a0d4"
Nov 03 08:11:22.992410 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:22.991245    5070 scope.go:122] "RemoveContainer" containerID="bb5ace5e28fb545b53a02d6e38b3007d317de568b9e88e5ef225ef916b2647fe"
Nov 03 08:11:23.005969 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:23.005950    5070 scope.go:122] "RemoveContainer" containerID="143ff4ac037a8e9eafe96b00feb9cdd0b4563cd4e9c68b6b97a9f76891191623"
Nov 03 08:11:23.019924 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:23.018221    5070 scope.go:122] "RemoveContainer" containerID="495058cc171927eb9632a6bb8dd928a3aa7cacd6fa99f34a71226695e7184afc"
Nov 03 08:11:23.030123 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:23.029053    5070 scope.go:122] "RemoveContainer" containerID="d1e89152c7c27f59e6a4261ed14cd3ff3022ec3e47a1f5f9d1ce2f7d19d457ad"
Nov 03 08:11:23.039010 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:23.038948    5070 scope.go:122] "RemoveContainer" containerID="dd9b7587b2e6dae86b36e6e29a5fc1bd8dd8a3bfd10c68cb0dd9de3899fb295a"
Nov 03 08:11:23.055041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:23.054923    5070 scope.go:122] "RemoveContainer" containerID="5e413b0b49e17818550abb8d1cd332516efbe398bce9e0ad5dbb906985ebb2e4"
Nov 03 08:11:25.027332 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:25.027302    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv"]
Nov 03 08:11:25.044943 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:25.044910    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3846835c-7mhrv"]
Nov 03 08:11:26.027352 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:26.027310    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a" path="/var/lib/kubelet/pods/dbdf6a0a-71b0-4ac6-a75a-0df538e27d5a/volumes"
Nov 03 08:11:31.223905 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:31.223871    5070 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:11:31.224379 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:31.223952    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f" podUID="157b7ff4-503c-4ecd-908c-2c078e875322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:11:34.506749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:34.506660    5070 generic.go:356] "Generic (PLEG): container finished" podID="157b7ff4-503c-4ecd-908c-2c078e875322" containerID="355dd9ad7c20d3e2267799e06e05f65b9b59d7b0d068944ac3522476d43805a9" exitCode=0
Nov 03 08:11:34.507306 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:34.507031    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f" event={"ID":"157b7ff4-503c-4ecd-908c-2c078e875322","Type":"ContainerDied","Data":"355dd9ad7c20d3e2267799e06e05f65b9b59d7b0d068944ac3522476d43805a9"}
Nov 03 08:11:34.507928 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:34.507626    5070 scope.go:122] "RemoveContainer" containerID="355dd9ad7c20d3e2267799e06e05f65b9b59d7b0d068944ac3522476d43805a9"
Nov 03 08:11:34.510249 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:34.510215    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:11:34.510359 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:34.510325    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:11:34.510400 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:34.510389    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:11:35.520598 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:35.520563    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f" event={"ID":"157b7ff4-503c-4ecd-908c-2c078e875322","Type":"ContainerStarted","Data":"64d96e4cc0fa351fa91788e6f817bb055b1d68119a26fe761ab5ae4859c830cf"}
Nov 03 08:11:35.881122 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:35.881081    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:11:35.881391 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:35.881328    5070 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 03 08:11:35.881465 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:35.881384    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f" podUID="157b7ff4-503c-4ecd-908c-2c078e875322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 03 08:11:36.028795 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:36.028746    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f"]
Nov 03 08:11:36.031279 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:36.031254    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6cf0844f-cgb4f"]
Nov 03 08:11:36.889247 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:36.889209    5070 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 03 08:11:36.889893 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:36.889857    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f" podUID="157b7ff4-503c-4ecd-908c-2c078e875322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 03 08:11:37.031389 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:37.031335    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6"]
Nov 03 08:11:37.037205 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:37.037183    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dd21693-qksdt"]
Nov 03 08:11:37.041631 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:37.041616    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-598c7ad0-zqkh6"]
Nov 03 08:11:37.045967 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:37.045947    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dd21693-qksdt"]
Nov 03 08:11:37.923272 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:37.923232    5070 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 03 08:11:37.923808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:37.923304    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f" podUID="157b7ff4-503c-4ecd-908c-2c078e875322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 03 08:11:38.029220 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:38.029148    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="323b9bca-191d-4d7e-8b2b-a41ec4be28a8" path="/var/lib/kubelet/pods/323b9bca-191d-4d7e-8b2b-a41ec4be28a8/volumes"
Nov 03 08:11:38.029774 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:38.029742    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2" path="/var/lib/kubelet/pods/6a1c1a14-1bc5-4ba5-94e8-9711aa7041a2/volumes"
Nov 03 08:11:38.030222 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:38.030205    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f66fab8d-b783-4223-8b36-edae53932323" path="/var/lib/kubelet/pods/f66fab8d-b783-4223-8b36-edae53932323/volumes"
Nov 03 08:11:38.030556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:38.030542    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1893a77-vlsl9"]
Nov 03 08:11:38.031334 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:38.031310    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1893a77-vlsl9"]
Nov 03 08:11:38.881177 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:38.881142    5070 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 03 08:11:38.881406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:38.881223    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f" podUID="157b7ff4-503c-4ecd-908c-2c078e875322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 03 08:11:39.881166 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:39.881119    5070 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Nov 03 08:11:39.881601 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:39.881187    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f" podUID="157b7ff4-503c-4ecd-908c-2c078e875322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Nov 03 08:11:40.028367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:40.028329    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77bb75c9-e45c-4c45-9242-a3269c5d6549" path="/var/lib/kubelet/pods/77bb75c9-e45c-4c45-9242-a3269c5d6549/volumes"
Nov 03 08:11:40.927331 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:40.927302    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:11:40.927923 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:40.927898    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:11:40.928651 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:40.928631    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:11:45.023989 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:45.023956    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58ce4c6e-xhg94"]
Nov 03 08:11:45.030170 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:45.030144    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58ce4c6e-xhg94"]
Nov 03 08:11:46.026950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:46.026914    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1703c4f9-5aea-4c19-8d85-d9040bc83baa" path="/var/lib/kubelet/pods/1703c4f9-5aea-4c19-8d85-d9040bc83baa/volumes"
Nov 03 08:11:46.853477 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:46.853442    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"]
Nov 03 08:11:46.854719 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:46.854688    5070 cpu_manager.go:414] "containerMap: removing container" podUID="158baf44-a9e1-4dba-ae8b-7a980fbd1443" containerName="revoke-keys"
Nov 03 08:11:46.854719 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:46.854719    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158baf44-a9e1-4dba-ae8b-7a980fbd1443" containerName="revoke-keys"
Nov 03 08:11:46.855001 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:46.854984    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="158baf44-a9e1-4dba-ae8b-7a980fbd1443" containerName="revoke-keys"
Nov 03 08:11:46.872230 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:46.872203    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:46.875524 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:46.875485    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-178e68-openshift-default-dockercfg-rg5th\""
Nov 03 08:11:46.875967 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:46.875914    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"]
Nov 03 08:11:46.985115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:46.980351    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istiod-ca-cert\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.004932 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.000423    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-token\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.012778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.005324    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9264d\" (UniqueName: \"kubernetes.io/projected/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-kube-api-access-9264d\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.013135 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.013108    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-podinfo\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.013348 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.013331    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-envoy\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.024778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.022825    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-workload-socket\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.024778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.022984    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-data\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.024778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.023022    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-credential-socket\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.024778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.023144    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-workload-certs\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.138990 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.138900    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9264d\" (UniqueName: \"kubernetes.io/projected/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-kube-api-access-9264d\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.140453 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.140030    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-podinfo\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.141165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.141142    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-envoy\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.141391 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.141366    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-workload-socket\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.141676 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.141657    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-data\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.141834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.141818    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-credential-socket\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.142075 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.142056    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-workload-certs\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.162785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.144617    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istiod-ca-cert\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.163047 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.163023    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-token\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.172392 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.143374    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-data\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.172603 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.143621    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-credential-socket\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.179782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.145544    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istiod-ca-cert\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.179994 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.142650    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-workload-socket\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.180113 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.156559    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-podinfo\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.180204 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.143029    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-workload-certs\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.204292 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.169351    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-envoy\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.204532 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.169993    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9264d\" (UniqueName: \"kubernetes.io/projected/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-kube-api-access-9264d\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.204650 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.177425    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-token\") pod \"e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") " pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.369357 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.369324    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"]
Nov 03 08:11:47.376677 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.376638    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mq5n\" (UniqueName: \"kubernetes.io/projected/fd8f468d-ba76-4155-85d8-7952b0cd31f5-kube-api-access-2mq5n\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.376856 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.376707    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-credential-socket\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.376958 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.376853    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-token\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.376958 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.376900    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-data\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.377064 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.377020    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-envoy\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.377118 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.377097    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istiod-ca-cert\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.377168 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.377128    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-podinfo\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.377168 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.377152    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-workload-certs\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.377268 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.377188    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-workload-socket\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.377419 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.377386    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.382883 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.381158    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e255cbee-openshift-default-dockercfg-tjfnm\""
Nov 03 08:11:47.397610 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.397531    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"]
Nov 03 08:11:47.496318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.485605    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mq5n\" (UniqueName: \"kubernetes.io/projected/fd8f468d-ba76-4155-85d8-7952b0cd31f5-kube-api-access-2mq5n\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.496318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.486006    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-credential-socket\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.496318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.486428    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-credential-socket\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.496318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.486780    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-token\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.496318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.486865    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-data\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.496318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.487239    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-data\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.496318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.487452    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-envoy\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.496318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.487948    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istiod-ca-cert\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.496318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.488909    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istiod-ca-cert\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.496318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.488980    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-podinfo\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.496318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.489020    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-workload-certs\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.496318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.489073    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-workload-socket\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.496318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.489428    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-workload-socket\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.496318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.490018    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-workload-certs\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.496318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.494382    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:47.496318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.495339    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-envoy\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.496318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.496244    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mq5n\" (UniqueName: \"kubernetes.io/projected/fd8f468d-ba76-4155-85d8-7952b0cd31f5-kube-api-access-2mq5n\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.497449 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.496409    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-podinfo\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.505339 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.503464    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-token\") pod \"e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") " pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:47.769490 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:47.769399    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:48.096564 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:11:48.096532    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11074ce5_9baa_4b3b_ae8b_b30409e5efbb.slice/crio-32f67b455818f6d12cca9a55a2d2ed278f21d26996f8f34ffd7f6d5dfe69f481 WatchSource:0}: Error finding container 32f67b455818f6d12cca9a55a2d2ed278f21d26996f8f34ffd7f6d5dfe69f481: Status 404 returned error can't find the container with id 32f67b455818f6d12cca9a55a2d2ed278f21d26996f8f34ffd7f6d5dfe69f481
Nov 03 08:11:48.100827 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:48.100141    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:11:48.100827 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:48.100239    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:11:48.100827 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:48.100282    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:11:48.121200 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:48.119953    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"]
Nov 03 08:11:48.254828 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:11:48.254741    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8f468d_ba76_4155_85d8_7952b0cd31f5.slice/crio-9057310183bfe9435fd56557cbdb8cf2aea31bf84527f58410da910067a8a37f WatchSource:0}: Error finding container 9057310183bfe9435fd56557cbdb8cf2aea31bf84527f58410da910067a8a37f: Status 404 returned error can't find the container with id 9057310183bfe9435fd56557cbdb8cf2aea31bf84527f58410da910067a8a37f
Nov 03 08:11:48.256894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:48.256132    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"]
Nov 03 08:11:48.260165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:48.260125    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:11:48.260367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:48.260345    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:11:48.260522 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:48.260496    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:11:48.711574 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:48.711541    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst" event={"ID":"11074ce5-9baa-4b3b-ae8b-b30409e5efbb","Type":"ContainerStarted","Data":"9a8b7c354325dac7f58d23b3467952b4983a5d2ac30a25c9c7642e085d01de75"}
Nov 03 08:11:48.711853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:48.711836    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst" event={"ID":"11074ce5-9baa-4b3b-ae8b-b30409e5efbb","Type":"ContainerStarted","Data":"32f67b455818f6d12cca9a55a2d2ed278f21d26996f8f34ffd7f6d5dfe69f481"}
Nov 03 08:11:48.714074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:48.714052    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx" event={"ID":"fd8f468d-ba76-4155-85d8-7952b0cd31f5","Type":"ContainerStarted","Data":"d584251cb2ab18dd63b570b1b17083630b0e090a483250046f95978238188b7b"}
Nov 03 08:11:48.714225 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:48.714213    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx" event={"ID":"fd8f468d-ba76-4155-85d8-7952b0cd31f5","Type":"ContainerStarted","Data":"9057310183bfe9435fd56557cbdb8cf2aea31bf84527f58410da910067a8a37f"}
Nov 03 08:11:48.732037 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:48.731987    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst" podStartSLOduration=2.731970523 podStartE2EDuration="2.731970523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:11:48.729262081 +0000 UTC m=+1106.843726013" watchObservedRunningTime="2026-11-03 08:11:48.731970523 +0000 UTC m=+1106.846434455"
Nov 03 08:11:48.751374 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:48.751320    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx" podStartSLOduration=1.7513096479999999 podStartE2EDuration="1.751309648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:11:48.746276068 +0000 UTC m=+1106.860739999" watchObservedRunningTime="2026-11-03 08:11:48.751309648 +0000 UTC m=+1106.865773632"
Nov 03 08:11:48.769932 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:48.769903    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:49.495120 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:49.495078    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:49.501164 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:49.501138    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:49.729206 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:49.729174    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:49.730509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:49.730483    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:11:49.777459 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:49.777380    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:50.581799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.581749    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"]
Nov 03 08:11:50.588271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.588244    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.597136 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.597114    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-dfb883-openshift-default-dockercfg-mtfb5\""
Nov 03 08:11:50.600464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.600435    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"]
Nov 03 08:11:50.625855 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.625829    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a77803-4112-45fe-bf9b-5ca069a5441f-istiod-ca-cert\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.626024 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.625903    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-envoy\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.626024 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.626006    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-token\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.626099 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.626087    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-credential-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.626206 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.626189    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-podinfo\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.626314 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.626290    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crp7h\" (UniqueName: \"kubernetes.io/projected/58a77803-4112-45fe-bf9b-5ca069a5441f-kube-api-access-crp7h\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.626403 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.626386    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-workload-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.626579 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.626559    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-data\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.626640 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.626626    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-workload-certs\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.728218 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.728189    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-credential-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.728414 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.728253    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-podinfo\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.728414 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.728306    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crp7h\" (UniqueName: \"kubernetes.io/projected/58a77803-4112-45fe-bf9b-5ca069a5441f-kube-api-access-crp7h\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.728414 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.728364    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-workload-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.728571 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.728513    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-data\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.728629 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.728584    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-workload-certs\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.728684 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.728652    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-credential-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.728735 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.728708    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a77803-4112-45fe-bf9b-5ca069a5441f-istiod-ca-cert\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.728844 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.728754    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-workload-socket\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.728844 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.728789    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-envoy\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.728844 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.728832    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-token\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.729013 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.728929    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-data\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.729013 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.728963    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-workload-certs\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.729619 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.729581    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a77803-4112-45fe-bf9b-5ca069a5441f-istiod-ca-cert\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.731155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.731134    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-envoy\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.731409 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.731386    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-podinfo\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.738390 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.738366    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:50.739235 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.739218    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:11:50.742477 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.742457    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-token\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.742644 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.742626    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crp7h\" (UniqueName: \"kubernetes.io/projected/58a77803-4112-45fe-bf9b-5ca069a5441f-kube-api-access-crp7h\") pod \"e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") " pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:50.939459 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:50.934222    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:51.490318 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:11:51.490276    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77803_4112_45fe_bf9b_5ca069a5441f.slice/crio-c09bf042184b873b27f9384a6cb6afa64e7b9feda426001cea74ab3b16c0080d WatchSource:0}: Error finding container c09bf042184b873b27f9384a6cb6afa64e7b9feda426001cea74ab3b16c0080d: Status 404 returned error can't find the container with id c09bf042184b873b27f9384a6cb6afa64e7b9feda426001cea74ab3b16c0080d
Nov 03 08:11:51.491814 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:51.491791    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"]
Nov 03 08:11:51.493708 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:51.493668    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:11:51.493798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:51.493767    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:11:51.493846 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:51.493813    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:11:51.750014 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:51.749973    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr" event={"ID":"58a77803-4112-45fe-bf9b-5ca069a5441f","Type":"ContainerStarted","Data":"0df6c92f2ed4ae98ac879da0364e52048aa4c49e89ea406ac019138d1ae36d7a"}
Nov 03 08:11:51.750622 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:51.750048    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr" event={"ID":"58a77803-4112-45fe-bf9b-5ca069a5441f","Type":"ContainerStarted","Data":"c09bf042184b873b27f9384a6cb6afa64e7b9feda426001cea74ab3b16c0080d"}
Nov 03 08:11:51.771047 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:51.770947    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr" podStartSLOduration=1.7709352219999999 podStartE2EDuration="1.770935222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:11:51.767816511 +0000 UTC m=+1109.882280442" watchObservedRunningTime="2026-11-03 08:11:51.770935222 +0000 UTC m=+1109.885399153"
Nov 03 08:11:51.970778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:51.965926    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:52.037121 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:52.037041    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6"]
Nov 03 08:11:52.045356 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:52.045301    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac62b195-kz2j6"]
Nov 03 08:11:52.073336 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:11:52.073290    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:11:52.978781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:52.978489    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:53.887053 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:53.886152    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:53.913183 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:53.910071    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:11:54.076053 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:54.076018    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48af7f66-65e9-4ef0-b99a-7046e3303bef" path="/var/lib/kubelet/pods/48af7f66-65e9-4ef0-b99a-7046e3303bef/volumes"
Nov 03 08:11:56.139110 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:11:56.139079    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:12:00.431787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:00.430908    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:12:00.455261 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:00.454814    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:12:00.470224 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:00.464590    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:12:00.677783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:00.664198    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:12:00.704845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:00.702851    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:12:11.042705 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:11.038250    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx"]
Nov 03 08:12:11.045846 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:11.045801    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b1fce47-phkkx"]
Nov 03 08:12:12.027547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:12.027516    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ab19328-5bd5-40cc-b030-af55b89d76ba" path="/var/lib/kubelet/pods/3ab19328-5bd5-40cc-b030-af55b89d76ba/volumes"
Nov 03 08:12:12.035389 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:12.035354    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw"]
Nov 03 08:12:12.043459 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:12.043439    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-408f9b14-m2mbw"]
Nov 03 08:12:12.339663 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:12.339588    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr"]
Nov 03 08:12:12.344496 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:12.344479    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr"
Nov 03 08:12:12.351140 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:12.351066    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:12:12.351260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:12.351170    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:12:12.351260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:12.351201    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:12:12.351865 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:12.351842    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:12:12.359665 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:12.359643    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr"]
Nov 03 08:12:12.436425 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:12.436387    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s64x7\" (UniqueName: \"kubernetes.io/projected/4f67019d-5a7a-49c5-a185-6706d3fe25c1-kube-api-access-s64x7\") pod \"maas-api-revoke-keys-e2e-mt-cb887217-4f8hr\" (UID: \"4f67019d-5a7a-49c5-a185-6706d3fe25c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr"
Nov 03 08:12:12.436710 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:12.436681    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f67019d-5a7a-49c5-a185-6706d3fe25c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb887217-4f8hr\" (UID: \"4f67019d-5a7a-49c5-a185-6706d3fe25c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr"
Nov 03 08:12:12.437796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:12.437769    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f67019d-5a7a-49c5-a185-6706d3fe25c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb887217-4f8hr\" (UID: \"4f67019d-5a7a-49c5-a185-6706d3fe25c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr"
Nov 03 08:12:12.539080 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:12.539049    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f67019d-5a7a-49c5-a185-6706d3fe25c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb887217-4f8hr\" (UID: \"4f67019d-5a7a-49c5-a185-6706d3fe25c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr"
Nov 03 08:12:12.539403 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:12.539382    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s64x7\" (UniqueName: \"kubernetes.io/projected/4f67019d-5a7a-49c5-a185-6706d3fe25c1-kube-api-access-s64x7\") pod \"maas-api-revoke-keys-e2e-mt-cb887217-4f8hr\" (UID: \"4f67019d-5a7a-49c5-a185-6706d3fe25c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr"
Nov 03 08:12:12.539501 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:12.539424    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f67019d-5a7a-49c5-a185-6706d3fe25c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb887217-4f8hr\" (UID: \"4f67019d-5a7a-49c5-a185-6706d3fe25c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr"
Nov 03 08:12:12.540086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:12.540068    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f67019d-5a7a-49c5-a185-6706d3fe25c1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb887217-4f8hr\" (UID: \"4f67019d-5a7a-49c5-a185-6706d3fe25c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr"
Nov 03 08:12:12.541565 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:12.541537    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f67019d-5a7a-49c5-a185-6706d3fe25c1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb887217-4f8hr\" (UID: \"4f67019d-5a7a-49c5-a185-6706d3fe25c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr"
Nov 03 08:12:12.548336 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:12.548310    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s64x7\" (UniqueName: \"kubernetes.io/projected/4f67019d-5a7a-49c5-a185-6706d3fe25c1-kube-api-access-s64x7\") pod \"maas-api-revoke-keys-e2e-mt-cb887217-4f8hr\" (UID: \"4f67019d-5a7a-49c5-a185-6706d3fe25c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr"
Nov 03 08:12:12.665532 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:12.665482    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr"
Nov 03 08:12:12.991902 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:12:12.991876    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f67019d_5a7a_49c5_a185_6706d3fe25c1.slice/crio-ef9786a7ebc857245d6f55b8419536c2014c09c001c48a063eac4bd26022efec WatchSource:0}: Error finding container ef9786a7ebc857245d6f55b8419536c2014c09c001c48a063eac4bd26022efec: Status 404 returned error can't find the container with id ef9786a7ebc857245d6f55b8419536c2014c09c001c48a063eac4bd26022efec
Nov 03 08:12:12.995823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:12.995798    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr"]
Nov 03 08:12:13.467107 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:13.466990    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr" event={"ID":"4f67019d-5a7a-49c5-a185-6706d3fe25c1","Type":"ContainerStarted","Data":"dc727e7e98197d4e51c0475c55df1de39f5abe6b7aefb14e94add299aa00faea"}
Nov 03 08:12:13.467107 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:13.467045    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr" event={"ID":"4f67019d-5a7a-49c5-a185-6706d3fe25c1","Type":"ContainerStarted","Data":"ef9786a7ebc857245d6f55b8419536c2014c09c001c48a063eac4bd26022efec"}
Nov 03 08:12:13.482882 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:13.482392    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr" podStartSLOduration=1.482375828 podStartE2EDuration="1.482375828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:12:13.481301041 +0000 UTC m=+1131.595764971" watchObservedRunningTime="2026-11-03 08:12:13.482375828 +0000 UTC m=+1131.596839760"
Nov 03 08:12:14.027725 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:14.027692    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb1f4c17-b03f-4c23-9742-1869aeb79837" path="/var/lib/kubelet/pods/bb1f4c17-b03f-4c23-9742-1869aeb79837/volumes"
Nov 03 08:12:14.028102 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:14.028086    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl"]
Nov 03 08:12:14.032873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:14.032851    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29cf07-7phcl"]
Nov 03 08:12:14.479437 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:14.479402    5070 generic.go:356] "Generic (PLEG): container finished" podID="4f67019d-5a7a-49c5-a185-6706d3fe25c1" containerID="dc727e7e98197d4e51c0475c55df1de39f5abe6b7aefb14e94add299aa00faea" exitCode=0
Nov 03 08:12:14.479844 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:14.479491    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr" event={"ID":"4f67019d-5a7a-49c5-a185-6706d3fe25c1","Type":"ContainerDied","Data":"dc727e7e98197d4e51c0475c55df1de39f5abe6b7aefb14e94add299aa00faea"}
Nov 03 08:12:15.768181 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:15.768119    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr"
Nov 03 08:12:15.898666 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:15.898639    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f67019d-5a7a-49c5-a185-6706d3fe25c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f67019d-5a7a-49c5-a185-6706d3fe25c1-maas-api-openshift-ca\") pod \"4f67019d-5a7a-49c5-a185-6706d3fe25c1\" (UID: \"4f67019d-5a7a-49c5-a185-6706d3fe25c1\") "
Nov 03 08:12:15.898881 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:15.898864    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f67019d-5a7a-49c5-a185-6706d3fe25c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f67019d-5a7a-49c5-a185-6706d3fe25c1-maas-api-xks-ca\") pod \"4f67019d-5a7a-49c5-a185-6706d3fe25c1\" (UID: \"4f67019d-5a7a-49c5-a185-6706d3fe25c1\") "
Nov 03 08:12:15.898939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:15.898924    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f67019d-5a7a-49c5-a185-6706d3fe25c1-kube-api-access-s64x7\" (UniqueName: \"kubernetes.io/projected/4f67019d-5a7a-49c5-a185-6706d3fe25c1-kube-api-access-s64x7\") pod \"4f67019d-5a7a-49c5-a185-6706d3fe25c1\" (UID: \"4f67019d-5a7a-49c5-a185-6706d3fe25c1\") "
Nov 03 08:12:15.899012 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:15.898988    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f67019d-5a7a-49c5-a185-6706d3fe25c1-maas-api-openshift-ca" pod "4f67019d-5a7a-49c5-a185-6706d3fe25c1" (UID: "4f67019d-5a7a-49c5-a185-6706d3fe25c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:12:15.899478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:15.899459    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f67019d-5a7a-49c5-a185-6706d3fe25c1-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:12:15.902062 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:15.902035    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f67019d-5a7a-49c5-a185-6706d3fe25c1-maas-api-xks-ca" pod "4f67019d-5a7a-49c5-a185-6706d3fe25c1" (UID: "4f67019d-5a7a-49c5-a185-6706d3fe25c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:12:15.902154 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:15.902076    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f67019d-5a7a-49c5-a185-6706d3fe25c1-kube-api-access-s64x7" pod "4f67019d-5a7a-49c5-a185-6706d3fe25c1" (UID: "4f67019d-5a7a-49c5-a185-6706d3fe25c1"). InnerVolumeSpecName "kube-api-access-s64x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:12:16.001047 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:16.001021    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f67019d-5a7a-49c5-a185-6706d3fe25c1-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:12:16.001047 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:16.001044    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s64x7\" (UniqueName: \"kubernetes.io/projected/4f67019d-5a7a-49c5-a185-6706d3fe25c1-kube-api-access-s64x7\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:12:16.029623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:16.029557    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a919ae58-9b50-4450-a523-1f64f89433c0" path="/var/lib/kubelet/pods/a919ae58-9b50-4450-a523-1f64f89433c0/volumes"
Nov 03 08:12:16.030113 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:16.030096    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2c0e3b3-6r9nz"]
Nov 03 08:12:16.032907 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:16.032886    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2c0e3b3-6r9nz"]
Nov 03 08:12:16.508536 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:16.508503    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr"
Nov 03 08:12:16.508735 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:16.508568    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr" event={"ID":"4f67019d-5a7a-49c5-a185-6706d3fe25c1","Type":"ContainerDied","Data":"ef9786a7ebc857245d6f55b8419536c2014c09c001c48a063eac4bd26022efec"}
Nov 03 08:12:16.508735 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:16.508621    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef9786a7ebc857245d6f55b8419536c2014c09c001c48a063eac4bd26022efec"
Nov 03 08:12:18.028472 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:18.028440    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b9ce1ef-4f78-4fdf-b846-e209df28cafe" path="/var/lib/kubelet/pods/9b9ce1ef-4f78-4fdf-b846-e209df28cafe/volumes"
Nov 03 08:12:18.028864 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:18.028809    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9"]
Nov 03 08:12:18.028912 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:18.028877    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77229df9-46vf9"]
Nov 03 08:12:20.034165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:20.034118    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ce8710b-6c3b-4d5a-9826-f77113dc6630" path="/var/lib/kubelet/pods/7ce8710b-6c3b-4d5a-9826-f77113dc6630/volumes"
Nov 03 08:12:22.051907 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:12:22.051876    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:12:23.325674 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:23.325640    5070 scope.go:122] "RemoveContainer" containerID="9beef623c3e4a2bb1b9d609c622ed186477965a0a040890d4d54774bc44dd571"
Nov 03 08:12:23.337427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:23.337408    5070 scope.go:122] "RemoveContainer" containerID="8b8343f74dbe597bc31a34d88260e3089465dfad99f8083d304909bb83979707"
Nov 03 08:12:23.347734 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:23.347715    5070 scope.go:122] "RemoveContainer" containerID="e21c0c531633ae4b960b8e773afd955ec5ecae6e32cf410900cf829e113b4280"
Nov 03 08:12:24.057279 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:24.057247    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db487e8-kcmw4"]
Nov 03 08:12:24.062672 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:24.062648    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8db487e8-kcmw4"]
Nov 03 08:12:26.046312 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:26.046277    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a769bcbf-c5e1-4a55-a47e-cc82372f982a" path="/var/lib/kubelet/pods/a769bcbf-c5e1-4a55-a47e-cc82372f982a/volumes"
Nov 03 08:12:26.046871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:26.046795    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g"]
Nov 03 08:12:26.047186 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:26.047156    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f133ab31-tf49g"]
Nov 03 08:12:28.031548 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:28.031495    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9d86f14-d2d5-4166-a8ab-14db3536afa4" path="/var/lib/kubelet/pods/a9d86f14-d2d5-4166-a8ab-14db3536afa4/volumes"
Nov 03 08:12:31.036078 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:31.036042    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp"]
Nov 03 08:12:31.044985 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:31.044959    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b7103d0-9jzfp"]
Nov 03 08:12:32.134097 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:32.134056    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27583ffd-7959-4158-a5c2-f523349cda90" path="/var/lib/kubelet/pods/27583ffd-7959-4158-a5c2-f523349cda90/volumes"
Nov 03 08:12:34.142613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:34.142576    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d326211f-bk7nw"]
Nov 03 08:12:34.198490 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:34.198452    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d4e3763-7j8cd"]
Nov 03 08:12:34.245137 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:34.245099    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d326211f-bk7nw"]
Nov 03 08:12:34.253276 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:34.253240    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d4e3763-7j8cd"]
Nov 03 08:12:36.069778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:36.061260    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bbb013d-3d06-48e6-8656-b86171b2edbf" path="/var/lib/kubelet/pods/5bbb013d-3d06-48e6-8656-b86171b2edbf/volumes"
Nov 03 08:12:36.069778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:36.062074    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f364abcd-f24f-493e-9962-d20498f7abcc" path="/var/lib/kubelet/pods/f364abcd-f24f-493e-9962-d20498f7abcc/volumes"
Nov 03 08:12:37.791947 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:37.788964    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b"]
Nov 03 08:12:37.793484 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:37.793456    5070 cpu_manager.go:414] "containerMap: removing container" podUID="4f67019d-5a7a-49c5-a185-6706d3fe25c1" containerName="revoke-keys"
Nov 03 08:12:37.793484 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:37.793486    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f67019d-5a7a-49c5-a185-6706d3fe25c1" containerName="revoke-keys"
Nov 03 08:12:37.793904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:37.793881    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f67019d-5a7a-49c5-a185-6706d3fe25c1" containerName="revoke-keys"
Nov 03 08:12:37.803470 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:37.803439    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b"]
Nov 03 08:12:37.803613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:37.803552    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b"
Nov 03 08:12:37.809259 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:37.809233    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:12:37.809404 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:37.809385    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:12:37.882099 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:37.882063    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:12:37.882592 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:37.882570    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:12:37.945955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:37.945924    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s56z7\" (UniqueName: \"kubernetes.io/projected/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034-kube-api-access-s56z7\") pod \"maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b\" (UID: \"4eec2b89-b438-4e5b-b4fa-ac92fcc0d034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b"
Nov 03 08:12:37.946121 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:37.946084    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b\" (UID: \"4eec2b89-b438-4e5b-b4fa-ac92fcc0d034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b"
Nov 03 08:12:37.946420 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:37.946392    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b\" (UID: \"4eec2b89-b438-4e5b-b4fa-ac92fcc0d034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b"
Nov 03 08:12:38.047875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:38.047793    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b\" (UID: \"4eec2b89-b438-4e5b-b4fa-ac92fcc0d034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b"
Nov 03 08:12:38.048024 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:38.047896    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s56z7\" (UniqueName: \"kubernetes.io/projected/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034-kube-api-access-s56z7\") pod \"maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b\" (UID: \"4eec2b89-b438-4e5b-b4fa-ac92fcc0d034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b"
Nov 03 08:12:38.048024 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:38.047979    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b\" (UID: \"4eec2b89-b438-4e5b-b4fa-ac92fcc0d034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b"
Nov 03 08:12:38.048660 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:38.048626    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b\" (UID: \"4eec2b89-b438-4e5b-b4fa-ac92fcc0d034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b"
Nov 03 08:12:38.050383 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:38.050359    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b\" (UID: \"4eec2b89-b438-4e5b-b4fa-ac92fcc0d034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b"
Nov 03 08:12:38.084786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:38.084438    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s56z7\" (UniqueName: \"kubernetes.io/projected/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034-kube-api-access-s56z7\") pod \"maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b\" (UID: \"4eec2b89-b438-4e5b-b4fa-ac92fcc0d034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b"
Nov 03 08:12:38.133011 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:38.132982    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b"
Nov 03 08:12:38.491133 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:38.491091    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b"]
Nov 03 08:12:39.388449 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:39.388415    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b" event={"ID":"4eec2b89-b438-4e5b-b4fa-ac92fcc0d034","Type":"ContainerStarted","Data":"10755bb0f367772a9d9d8052a1c073ea3fc121f79ca584fb07b84da2be465b8a"}
Nov 03 08:12:39.388948 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:39.388477    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b" event={"ID":"4eec2b89-b438-4e5b-b4fa-ac92fcc0d034","Type":"ContainerStarted","Data":"78d50ba5a9a20c50db43e1899173fa0aa52f244b68d3642be30a7fc44cd88026"}
Nov 03 08:12:39.418122 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:39.418065    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b" podStartSLOduration=2.418050645 podStartE2EDuration="2.418050645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:12:39.414789157 +0000 UTC m=+1157.529253088" watchObservedRunningTime="2026-11-03 08:12:39.418050645 +0000 UTC m=+1157.532514591"
Nov 03 08:12:40.110780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:40.102820    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cf09e14-nm7h8"]
Nov 03 08:12:40.136877 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:40.131407    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cf09e14-nm7h8"]
Nov 03 08:12:40.401043 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:40.400962    5070 generic.go:356] "Generic (PLEG): container finished" podID="4eec2b89-b438-4e5b-b4fa-ac92fcc0d034" containerID="10755bb0f367772a9d9d8052a1c073ea3fc121f79ca584fb07b84da2be465b8a" exitCode=0
Nov 03 08:12:40.401427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:40.401052    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b" event={"ID":"4eec2b89-b438-4e5b-b4fa-ac92fcc0d034","Type":"ContainerDied","Data":"10755bb0f367772a9d9d8052a1c073ea3fc121f79ca584fb07b84da2be465b8a"}
Nov 03 08:12:41.069959 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:41.069886    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6"]
Nov 03 08:12:41.077400 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:41.077364    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f48a026a-8nzm6"]
Nov 03 08:12:41.918770 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:41.918719    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b"
Nov 03 08:12:41.979573 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:41.979517    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034-maas-api-openshift-ca\") pod \"4eec2b89-b438-4e5b-b4fa-ac92fcc0d034\" (UID: \"4eec2b89-b438-4e5b-b4fa-ac92fcc0d034\") "
Nov 03 08:12:41.979926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:41.979585    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034-kube-api-access-s56z7\" (UniqueName: \"kubernetes.io/projected/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034-kube-api-access-s56z7\") pod \"4eec2b89-b438-4e5b-b4fa-ac92fcc0d034\" (UID: \"4eec2b89-b438-4e5b-b4fa-ac92fcc0d034\") "
Nov 03 08:12:41.979926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:41.979711    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034-maas-api-xks-ca\") pod \"4eec2b89-b438-4e5b-b4fa-ac92fcc0d034\" (UID: \"4eec2b89-b438-4e5b-b4fa-ac92fcc0d034\") "
Nov 03 08:12:41.986499 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:41.986033    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034-maas-api-openshift-ca" pod "4eec2b89-b438-4e5b-b4fa-ac92fcc0d034" (UID: "4eec2b89-b438-4e5b-b4fa-ac92fcc0d034"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:12:41.999547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:41.999462    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034-kube-api-access-s56z7" pod "4eec2b89-b438-4e5b-b4fa-ac92fcc0d034" (UID: "4eec2b89-b438-4e5b-b4fa-ac92fcc0d034"). InnerVolumeSpecName "kube-api-access-s56z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:12:41.999701 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:41.999590    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034-maas-api-xks-ca" pod "4eec2b89-b438-4e5b-b4fa-ac92fcc0d034" (UID: "4eec2b89-b438-4e5b-b4fa-ac92fcc0d034"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:12:42.067456 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:42.067418    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3acd647f-bec3-4ace-8674-2c7009f9e91c" path="/var/lib/kubelet/pods/3acd647f-bec3-4ace-8674-2c7009f9e91c/volumes"
Nov 03 08:12:42.068397 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:42.068369    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d7b0574-49d7-4b35-94cb-065d525d881a" path="/var/lib/kubelet/pods/3d7b0574-49d7-4b35-94cb-065d525d881a/volumes"
Nov 03 08:12:42.069298 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:42.069279    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-425b2345-ctlh2"]
Nov 03 08:12:42.069442 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:42.069429    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-425b2345-ctlh2"]
Nov 03 08:12:42.083007 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:42.082984    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:12:42.083098 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:42.083019    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s56z7\" (UniqueName: \"kubernetes.io/projected/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034-kube-api-access-s56z7\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:12:42.083098 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:42.083033    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:12:42.423958 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:42.423928    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b"
Nov 03 08:12:42.424258 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:42.423964    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b" event={"ID":"4eec2b89-b438-4e5b-b4fa-ac92fcc0d034","Type":"ContainerDied","Data":"78d50ba5a9a20c50db43e1899173fa0aa52f244b68d3642be30a7fc44cd88026"}
Nov 03 08:12:42.424258 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:42.424023    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78d50ba5a9a20c50db43e1899173fa0aa52f244b68d3642be30a7fc44cd88026"
Nov 03 08:12:43.038169 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:43.038121    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3db6d499-75l5p"]
Nov 03 08:12:43.043350 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:43.043303    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3db6d499-75l5p"]
Nov 03 08:12:44.029312 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:44.029268    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204a59ad-3b92-4bc7-ad14-2dc39aa8a104" path="/var/lib/kubelet/pods/204a59ad-3b92-4bc7-ad14-2dc39aa8a104/volumes"
Nov 03 08:12:44.029794 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:44.029743    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="588587e7-5d9a-41cf-baa4-1d998c96a47e" path="/var/lib/kubelet/pods/588587e7-5d9a-41cf-baa4-1d998c96a47e/volumes"
Nov 03 08:12:45.338619 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:45.338586    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml"]
Nov 03 08:12:45.339393 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:45.339376    5070 cpu_manager.go:414] "containerMap: removing container" podUID="4eec2b89-b438-4e5b-b4fa-ac92fcc0d034" containerName="revoke-keys"
Nov 03 08:12:45.339439 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:45.339394    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eec2b89-b438-4e5b-b4fa-ac92fcc0d034" containerName="revoke-keys"
Nov 03 08:12:45.339547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:45.339538    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eec2b89-b438-4e5b-b4fa-ac92fcc0d034" containerName="revoke-keys"
Nov 03 08:12:45.375332 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:45.375294    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml"]
Nov 03 08:12:45.375523 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:45.375429    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml"
Nov 03 08:12:45.380221 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:45.380193    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:12:45.380489 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:45.380441    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:12:45.381024 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:45.380991    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:12:45.381117 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:45.381061    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:12:45.466729 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:45.466699    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d3ca38f-b7bc-431d-a1f9-aa69afe33648-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d563d08-pqgml\" (UID: \"5d3ca38f-b7bc-431d-a1f9-aa69afe33648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml"
Nov 03 08:12:45.466915 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:45.466868    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d3ca38f-b7bc-431d-a1f9-aa69afe33648-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d563d08-pqgml\" (UID: \"5d3ca38f-b7bc-431d-a1f9-aa69afe33648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml"
Nov 03 08:12:45.467015 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:45.466997    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52mqk\" (UniqueName: \"kubernetes.io/projected/5d3ca38f-b7bc-431d-a1f9-aa69afe33648-kube-api-access-52mqk\") pod \"maas-api-revoke-sub-1d563d08-pqgml\" (UID: \"5d3ca38f-b7bc-431d-a1f9-aa69afe33648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml"
Nov 03 08:12:45.596890 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:45.585208    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d3ca38f-b7bc-431d-a1f9-aa69afe33648-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d563d08-pqgml\" (UID: \"5d3ca38f-b7bc-431d-a1f9-aa69afe33648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml"
Nov 03 08:12:45.597960 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:45.597859    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d3ca38f-b7bc-431d-a1f9-aa69afe33648-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d563d08-pqgml\" (UID: \"5d3ca38f-b7bc-431d-a1f9-aa69afe33648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml"
Nov 03 08:12:45.599410 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:45.599376    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d3ca38f-b7bc-431d-a1f9-aa69afe33648-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d563d08-pqgml\" (UID: \"5d3ca38f-b7bc-431d-a1f9-aa69afe33648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml"
Nov 03 08:12:45.600247 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:45.600080    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52mqk\" (UniqueName: \"kubernetes.io/projected/5d3ca38f-b7bc-431d-a1f9-aa69afe33648-kube-api-access-52mqk\") pod \"maas-api-revoke-sub-1d563d08-pqgml\" (UID: \"5d3ca38f-b7bc-431d-a1f9-aa69afe33648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml"
Nov 03 08:12:45.617780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:45.613000    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d3ca38f-b7bc-431d-a1f9-aa69afe33648-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d563d08-pqgml\" (UID: \"5d3ca38f-b7bc-431d-a1f9-aa69afe33648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml"
Nov 03 08:12:45.646811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:45.635589    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52mqk\" (UniqueName: \"kubernetes.io/projected/5d3ca38f-b7bc-431d-a1f9-aa69afe33648-kube-api-access-52mqk\") pod \"maas-api-revoke-sub-1d563d08-pqgml\" (UID: \"5d3ca38f-b7bc-431d-a1f9-aa69afe33648\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml"
Nov 03 08:12:45.806884 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:45.806847    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml"
Nov 03 08:12:46.474110 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:12:46.401408    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3ca38f_b7bc_431d_a1f9_aa69afe33648.slice/crio-05db91514871fd84e9e2e53d122f17d97e603510a6553c79654dab55ad7fc5d8 WatchSource:0}: Error finding container 05db91514871fd84e9e2e53d122f17d97e603510a6553c79654dab55ad7fc5d8: Status 404 returned error can't find the container with id 05db91514871fd84e9e2e53d122f17d97e603510a6553c79654dab55ad7fc5d8
Nov 03 08:12:46.493168 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:46.492059    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml"]
Nov 03 08:12:46.507882 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:46.507793    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml" event={"ID":"5d3ca38f-b7bc-431d-a1f9-aa69afe33648","Type":"ContainerStarted","Data":"05db91514871fd84e9e2e53d122f17d97e603510a6553c79654dab55ad7fc5d8"}
Nov 03 08:12:47.547878 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:47.547794    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml" event={"ID":"5d3ca38f-b7bc-431d-a1f9-aa69afe33648","Type":"ContainerStarted","Data":"53a2cba5156777a60e8630d41bc3616a2ff09c9228839982721322597cc4768b"}
Nov 03 08:12:47.598011 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:47.597947    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml" podStartSLOduration=2.597926354 podStartE2EDuration="2.597926354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:12:47.597253893 +0000 UTC m=+1165.711717817" watchObservedRunningTime="2026-11-03 08:12:47.597926354 +0000 UTC m=+1165.712390286"
Nov 03 08:12:48.619208 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:48.619147    5070 generic.go:356] "Generic (PLEG): container finished" podID="5d3ca38f-b7bc-431d-a1f9-aa69afe33648" containerID="53a2cba5156777a60e8630d41bc3616a2ff09c9228839982721322597cc4768b" exitCode=0
Nov 03 08:12:48.620257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:48.619501    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml" event={"ID":"5d3ca38f-b7bc-431d-a1f9-aa69afe33648","Type":"ContainerDied","Data":"53a2cba5156777a60e8630d41bc3616a2ff09c9228839982721322597cc4768b"}
Nov 03 08:12:50.297075 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.297043    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt"]
Nov 03 08:12:50.312400 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.312374    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt"]
Nov 03 08:12:50.312514 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.312463    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt"
Nov 03 08:12:50.317063 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.316996    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml"
Nov 03 08:12:50.424461 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.424380    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d3ca38f-b7bc-431d-a1f9-aa69afe33648-kube-api-access-52mqk\" (UniqueName: \"kubernetes.io/projected/5d3ca38f-b7bc-431d-a1f9-aa69afe33648-kube-api-access-52mqk\") pod \"5d3ca38f-b7bc-431d-a1f9-aa69afe33648\" (UID: \"5d3ca38f-b7bc-431d-a1f9-aa69afe33648\") "
Nov 03 08:12:50.424671 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.424652    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d3ca38f-b7bc-431d-a1f9-aa69afe33648-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d3ca38f-b7bc-431d-a1f9-aa69afe33648-maas-api-openshift-ca\") pod \"5d3ca38f-b7bc-431d-a1f9-aa69afe33648\" (UID: \"5d3ca38f-b7bc-431d-a1f9-aa69afe33648\") "
Nov 03 08:12:50.424872 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.424851    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d3ca38f-b7bc-431d-a1f9-aa69afe33648-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d3ca38f-b7bc-431d-a1f9-aa69afe33648-maas-api-xks-ca\") pod \"5d3ca38f-b7bc-431d-a1f9-aa69afe33648\" (UID: \"5d3ca38f-b7bc-431d-a1f9-aa69afe33648\") "
Nov 03 08:12:50.425195 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.425169    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d3ca38f-b7bc-431d-a1f9-aa69afe33648-maas-api-openshift-ca" pod "5d3ca38f-b7bc-431d-a1f9-aa69afe33648" (UID: "5d3ca38f-b7bc-431d-a1f9-aa69afe33648"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:12:50.425300 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.425276    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5f2321b-ee05-4de7-a27a-0e49d3102330-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a37b13ea-dtglt\" (UID: \"a5f2321b-ee05-4de7-a27a-0e49d3102330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt"
Nov 03 08:12:50.425438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.425419    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2jjn\" (UniqueName: \"kubernetes.io/projected/a5f2321b-ee05-4de7-a27a-0e49d3102330-kube-api-access-w2jjn\") pod \"maas-api-revoke-sub-a37b13ea-dtglt\" (UID: \"a5f2321b-ee05-4de7-a27a-0e49d3102330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt"
Nov 03 08:12:50.425665 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.425642    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5f2321b-ee05-4de7-a27a-0e49d3102330-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a37b13ea-dtglt\" (UID: \"a5f2321b-ee05-4de7-a27a-0e49d3102330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt"
Nov 03 08:12:50.425829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.425798    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d3ca38f-b7bc-431d-a1f9-aa69afe33648-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:12:50.492247 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.492181    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d3ca38f-b7bc-431d-a1f9-aa69afe33648-maas-api-xks-ca" pod "5d3ca38f-b7bc-431d-a1f9-aa69afe33648" (UID: "5d3ca38f-b7bc-431d-a1f9-aa69afe33648"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:12:50.493556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.493484    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3ca38f-b7bc-431d-a1f9-aa69afe33648-kube-api-access-52mqk" pod "5d3ca38f-b7bc-431d-a1f9-aa69afe33648" (UID: "5d3ca38f-b7bc-431d-a1f9-aa69afe33648"). InnerVolumeSpecName "kube-api-access-52mqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:12:50.555791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.544979    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5f2321b-ee05-4de7-a27a-0e49d3102330-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a37b13ea-dtglt\" (UID: \"a5f2321b-ee05-4de7-a27a-0e49d3102330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt"
Nov 03 08:12:50.556254 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.556218    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2jjn\" (UniqueName: \"kubernetes.io/projected/a5f2321b-ee05-4de7-a27a-0e49d3102330-kube-api-access-w2jjn\") pod \"maas-api-revoke-sub-a37b13ea-dtglt\" (UID: \"a5f2321b-ee05-4de7-a27a-0e49d3102330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt"
Nov 03 08:12:50.556594 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.556576    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5f2321b-ee05-4de7-a27a-0e49d3102330-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a37b13ea-dtglt\" (UID: \"a5f2321b-ee05-4de7-a27a-0e49d3102330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt"
Nov 03 08:12:50.567780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.557006    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d3ca38f-b7bc-431d-a1f9-aa69afe33648-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:12:50.568342 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.568312    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52mqk\" (UniqueName: \"kubernetes.io/projected/5d3ca38f-b7bc-431d-a1f9-aa69afe33648-kube-api-access-52mqk\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:12:50.568494 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.558876    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5f2321b-ee05-4de7-a27a-0e49d3102330-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a37b13ea-dtglt\" (UID: \"a5f2321b-ee05-4de7-a27a-0e49d3102330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt"
Nov 03 08:12:50.588224 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.588157    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5f2321b-ee05-4de7-a27a-0e49d3102330-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a37b13ea-dtglt\" (UID: \"a5f2321b-ee05-4de7-a27a-0e49d3102330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt"
Nov 03 08:12:50.629793 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.606428    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2jjn\" (UniqueName: \"kubernetes.io/projected/a5f2321b-ee05-4de7-a27a-0e49d3102330-kube-api-access-w2jjn\") pod \"maas-api-revoke-sub-a37b13ea-dtglt\" (UID: \"a5f2321b-ee05-4de7-a27a-0e49d3102330\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt"
Nov 03 08:12:50.671806 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.657016    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt"
Nov 03 08:12:50.701837 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.694246    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml" event={"ID":"5d3ca38f-b7bc-431d-a1f9-aa69afe33648","Type":"ContainerDied","Data":"05db91514871fd84e9e2e53d122f17d97e603510a6553c79654dab55ad7fc5d8"}
Nov 03 08:12:50.701837 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.694310    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05db91514871fd84e9e2e53d122f17d97e603510a6553c79654dab55ad7fc5d8"
Nov 03 08:12:50.701837 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:50.694375    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml"
Nov 03 08:12:51.122141 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:12:51.122107    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f2321b_ee05_4de7_a27a_0e49d3102330.slice/crio-da0b0f8cf86950af94ccbe25c8499cb7a5d062f91ba18300f49520b0b38b34b0 WatchSource:0}: Error finding container da0b0f8cf86950af94ccbe25c8499cb7a5d062f91ba18300f49520b0b38b34b0: Status 404 returned error can't find the container with id da0b0f8cf86950af94ccbe25c8499cb7a5d062f91ba18300f49520b0b38b34b0
Nov 03 08:12:51.122543 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:51.122519    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt"]
Nov 03 08:12:51.798700 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:51.798568    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt" event={"ID":"a5f2321b-ee05-4de7-a27a-0e49d3102330","Type":"ContainerStarted","Data":"da0b0f8cf86950af94ccbe25c8499cb7a5d062f91ba18300f49520b0b38b34b0"}
Nov 03 08:12:51.805865 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:51.802869    5070 patch_prober.go:29] interesting pod/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:12:51.805865 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:51.802929    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx" podUID="fd8f468d-ba76-4155-85d8-7952b0cd31f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:12:52.550199 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:12:52.550160    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:12:52.811402 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:52.811326    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt" event={"ID":"a5f2321b-ee05-4de7-a27a-0e49d3102330","Type":"ContainerStarted","Data":"fc07b5508d29a7ef97c22b7ee29b93d8468994652e554ac321024055707cfa3c"}
Nov 03 08:12:52.836347 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:52.836299    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt" podStartSLOduration=2.83628486 podStartE2EDuration="2.83628486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:12:52.831906145 +0000 UTC m=+1170.946370079" watchObservedRunningTime="2026-11-03 08:12:52.83628486 +0000 UTC m=+1170.950748791"
Nov 03 08:12:53.822295 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:53.822259    5070 generic.go:356] "Generic (PLEG): container finished" podID="fd8f468d-ba76-4155-85d8-7952b0cd31f5" containerID="d584251cb2ab18dd63b570b1b17083630b0e090a483250046f95978238188b7b" exitCode=0
Nov 03 08:12:53.822731 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:53.822329    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx" event={"ID":"fd8f468d-ba76-4155-85d8-7952b0cd31f5","Type":"ContainerDied","Data":"d584251cb2ab18dd63b570b1b17083630b0e090a483250046f95978238188b7b"}
Nov 03 08:12:53.822843 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:53.822822    5070 scope.go:122] "RemoveContainer" containerID="d584251cb2ab18dd63b570b1b17083630b0e090a483250046f95978238188b7b"
Nov 03 08:12:53.824478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:53.824456    5070 generic.go:356] "Generic (PLEG): container finished" podID="a5f2321b-ee05-4de7-a27a-0e49d3102330" containerID="fc07b5508d29a7ef97c22b7ee29b93d8468994652e554ac321024055707cfa3c" exitCode=0
Nov 03 08:12:53.824578 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:53.824555    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt" event={"ID":"a5f2321b-ee05-4de7-a27a-0e49d3102330","Type":"ContainerDied","Data":"fc07b5508d29a7ef97c22b7ee29b93d8468994652e554ac321024055707cfa3c"}
Nov 03 08:12:53.825231 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:53.825201    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:12:53.825301 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:53.825287    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:12:53.825350 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:53.825336    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:12:54.838367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:54.838323    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx" event={"ID":"fd8f468d-ba76-4155-85d8-7952b0cd31f5","Type":"ContainerStarted","Data":"4bfb76d6582f248ba0b56bdb29e83acc63f19532dd9f8cad7ab54f0462f6e212"}
Nov 03 08:12:55.026362 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:55.026329    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6"]
Nov 03 08:12:55.046427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:55.046391    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc"]
Nov 03 08:12:55.055359 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:55.055324    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54bc7452-ztkk6"]
Nov 03 08:12:55.094199 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:55.094163    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d07a8f1f-f86hc"]
Nov 03 08:12:55.127554 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:55.127527    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt"
Nov 03 08:12:55.237463 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:55.237428    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f2321b-ee05-4de7-a27a-0e49d3102330-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5f2321b-ee05-4de7-a27a-0e49d3102330-maas-api-openshift-ca\") pod \"a5f2321b-ee05-4de7-a27a-0e49d3102330\" (UID: \"a5f2321b-ee05-4de7-a27a-0e49d3102330\") "
Nov 03 08:12:55.237625 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:55.237603    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5f2321b-ee05-4de7-a27a-0e49d3102330-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5f2321b-ee05-4de7-a27a-0e49d3102330-maas-api-xks-ca\") pod \"a5f2321b-ee05-4de7-a27a-0e49d3102330\" (UID: \"a5f2321b-ee05-4de7-a27a-0e49d3102330\") "
Nov 03 08:12:55.237706 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:55.237690    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f2321b-ee05-4de7-a27a-0e49d3102330-kube-api-access-w2jjn\" (UniqueName: \"kubernetes.io/projected/a5f2321b-ee05-4de7-a27a-0e49d3102330-kube-api-access-w2jjn\") pod \"a5f2321b-ee05-4de7-a27a-0e49d3102330\" (UID: \"a5f2321b-ee05-4de7-a27a-0e49d3102330\") "
Nov 03 08:12:55.237803 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:55.237778    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f2321b-ee05-4de7-a27a-0e49d3102330-maas-api-openshift-ca" pod "a5f2321b-ee05-4de7-a27a-0e49d3102330" (UID: "a5f2321b-ee05-4de7-a27a-0e49d3102330"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:12:55.238523 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:55.238497    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5f2321b-ee05-4de7-a27a-0e49d3102330-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:12:55.239720 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:55.239689    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f2321b-ee05-4de7-a27a-0e49d3102330-maas-api-xks-ca" pod "a5f2321b-ee05-4de7-a27a-0e49d3102330" (UID: "a5f2321b-ee05-4de7-a27a-0e49d3102330"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:12:55.239927 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:55.239907    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f2321b-ee05-4de7-a27a-0e49d3102330-kube-api-access-w2jjn" pod "a5f2321b-ee05-4de7-a27a-0e49d3102330" (UID: "a5f2321b-ee05-4de7-a27a-0e49d3102330"). InnerVolumeSpecName "kube-api-access-w2jjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:12:55.343293 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:55.343057    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2jjn\" (UniqueName: \"kubernetes.io/projected/a5f2321b-ee05-4de7-a27a-0e49d3102330-kube-api-access-w2jjn\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:12:55.343293 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:55.343099    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5f2321b-ee05-4de7-a27a-0e49d3102330-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:12:55.781820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:55.781171    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:12:55.781820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:55.781372    5070 patch_prober.go:29] interesting pod/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 03 08:12:55.781820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:55.781453    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx" podUID="fd8f468d-ba76-4155-85d8-7952b0cd31f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 03 08:12:55.848679 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:55.848655    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt"
Nov 03 08:12:55.849092 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:55.848726    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt" event={"ID":"a5f2321b-ee05-4de7-a27a-0e49d3102330","Type":"ContainerDied","Data":"da0b0f8cf86950af94ccbe25c8499cb7a5d062f91ba18300f49520b0b38b34b0"}
Nov 03 08:12:55.849092 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:55.848801    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da0b0f8cf86950af94ccbe25c8499cb7a5d062f91ba18300f49520b0b38b34b0"
Nov 03 08:12:56.028016 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:56.027980    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b91e763-0b36-4d03-bc1e-ccc40d9be701" path="/var/lib/kubelet/pods/7b91e763-0b36-4d03-bc1e-ccc40d9be701/volumes"
Nov 03 08:12:56.028520 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:56.028505    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6f1097b-a4d2-49fc-9db1-05721fed660d" path="/var/lib/kubelet/pods/b6f1097b-a4d2-49fc-9db1-05721fed660d/volumes"
Nov 03 08:12:56.282788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:56.277632    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:12:56.772190 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:56.772154    5070 patch_prober.go:29] interesting pod/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 03 08:12:56.772460 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:56.772431    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx" podUID="fd8f468d-ba76-4155-85d8-7952b0cd31f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 03 08:12:57.771150 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:57.771117    5070 patch_prober.go:29] interesting pod/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 03 08:12:57.771827 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:57.771793    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx" podUID="fd8f468d-ba76-4155-85d8-7952b0cd31f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 03 08:12:58.037414 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:58.037335    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr"]
Nov 03 08:12:58.043896 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:58.043854    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw"]
Nov 03 08:12:58.049289 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:58.049232    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e46eb63-ff7pr"]
Nov 03 08:12:58.054808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:58.054728    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-192e4d3a-k5xdw"]
Nov 03 08:12:58.782532 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:58.782095    5070 patch_prober.go:29] interesting pod/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 03 08:12:58.782532 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:58.782160    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx" podUID="fd8f468d-ba76-4155-85d8-7952b0cd31f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 03 08:12:59.775787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:59.773133    5070 patch_prober.go:29] interesting pod/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body=
Nov 03 08:12:59.775787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:12:59.773195    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx" podUID="fd8f468d-ba76-4155-85d8-7952b0cd31f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused"
Nov 03 08:13:00.031004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:00.030925    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d37485a-b563-4ff7-83e1-decd60b006f7" path="/var/lib/kubelet/pods/3d37485a-b563-4ff7-83e1-decd60b006f7/volumes"
Nov 03 08:13:00.031515 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:00.031497    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb98918b-b331-464e-88d2-081c8dcb7807" path="/var/lib/kubelet/pods/bb98918b-b331-464e-88d2-081c8dcb7807/volumes"
Nov 03 08:13:00.775507 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:00.775477    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:13:00.776001 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:00.775977    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:13:00.776737 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:00.776716    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:13:03.090157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:03.090121    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g"]
Nov 03 08:13:03.097742 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:03.097716    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf71ce56-zn65g"]
Nov 03 08:13:04.028223 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:04.028189    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1653421-2455-40e3-9103-f512bb881f04" path="/var/lib/kubelet/pods/e1653421-2455-40e3-9103-f512bb881f04/volumes"
Nov 03 08:13:04.136138 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:04.136094    5070 patch_prober.go:29] interesting pod/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:13:04.136981 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:04.136930    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng" podUID="87e84029-163b-4605-8c14-8da36b5c6cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:13:06.041092 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:06.041058    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2"]
Nov 03 08:13:06.050874 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:06.050849    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdb5bb3f-bn2f2"]
Nov 03 08:13:08.030244 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:08.030212    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a67dfc8-06fa-4be2-9be2-40477be1c36c" path="/var/lib/kubelet/pods/3a67dfc8-06fa-4be2-9be2-40477be1c36c/volumes"
Nov 03 08:13:09.326453 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:09.326414    5070 generic.go:356] "Generic (PLEG): container finished" podID="87e84029-163b-4605-8c14-8da36b5c6cf5" containerID="805447d0005e492d53887463bb38cc18474ffbe1a7a32d179f76767976b14712" exitCode=0
Nov 03 08:13:09.327223 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:09.327194    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng" event={"ID":"87e84029-163b-4605-8c14-8da36b5c6cf5","Type":"ContainerDied","Data":"805447d0005e492d53887463bb38cc18474ffbe1a7a32d179f76767976b14712"}
Nov 03 08:13:09.337842 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:09.327922    5070 scope.go:122] "RemoveContainer" containerID="805447d0005e492d53887463bb38cc18474ffbe1a7a32d179f76767976b14712"
Nov 03 08:13:09.352015 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:09.351971    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:13:09.377980 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:09.377933    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:13:09.378317 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:09.378286    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:13:09.406302 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:09.400867    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr"]
Nov 03 08:13:09.416057 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:09.414446    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-4f8hr"]
Nov 03 08:13:10.027941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:10.027908    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f67019d-5a7a-49c5-a185-6706d3fe25c1" path="/var/lib/kubelet/pods/4f67019d-5a7a-49c5-a185-6706d3fe25c1/volumes"
Nov 03 08:13:10.338877 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:10.338779    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng" event={"ID":"87e84029-163b-4605-8c14-8da36b5c6cf5","Type":"ContainerStarted","Data":"1570717c24ba0cf1971c9cf77416c998916652eff83a3ed7300b8cf95145c14a"}
Nov 03 08:13:10.539200 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:10.539163    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:13:10.539384 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:10.539323    5070 patch_prober.go:29] interesting pod/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 03 08:13:10.539440 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:10.539375    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng" podUID="87e84029-163b-4605-8c14-8da36b5c6cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 03 08:13:11.548795 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:11.544319    5070 patch_prober.go:29] interesting pod/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 03 08:13:11.548795 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:11.544379    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng" podUID="87e84029-163b-4605-8c14-8da36b5c6cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 03 08:13:12.539318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:12.539279    5070 patch_prober.go:29] interesting pod/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 03 08:13:12.539496 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:12.539351    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng" podUID="87e84029-163b-4605-8c14-8da36b5c6cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 03 08:13:13.551784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:13.539214    5070 patch_prober.go:29] interesting pod/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 03 08:13:13.551784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:13.539291    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng" podUID="87e84029-163b-4605-8c14-8da36b5c6cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 03 08:13:14.538556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:14.538512    5070 patch_prober.go:29] interesting pod/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 03 08:13:14.538885 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:14.538853    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng" podUID="87e84029-163b-4605-8c14-8da36b5c6cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 03 08:13:15.538552 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:15.538511    5070 patch_prober.go:29] interesting pod/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body=
Nov 03 08:13:15.539004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:15.538588    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng" podUID="87e84029-163b-4605-8c14-8da36b5c6cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused"
Nov 03 08:13:16.544385 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:16.544357    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:13:16.544876 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:16.544852    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:13:16.545606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:16.545587    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:13:22.057879 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:13:22.057842    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:13:22.333589 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:22.333509    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/2.log"
Nov 03 08:13:22.334194 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:22.334171    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/2.log"
Nov 03 08:13:23.419097 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:23.419068    5070 scope.go:122] "RemoveContainer" containerID="0719a67acecd83c256f950d076472b200421777355d3c17a05eb7fc98619970c"
Nov 03 08:13:23.431262 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:23.431242    5070 scope.go:122] "RemoveContainer" containerID="42239431bc9311b7c1ad6a351832007fafa7de84172bd8f45f8322498c56a319"
Nov 03 08:13:23.445518 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:23.443787    5070 scope.go:122] "RemoveContainer" containerID="68c58bc4981f62f88dda731953e87637bbe5a96065fdcc9320558bab97193543"
Nov 03 08:13:23.461836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:23.461811    5070 scope.go:122] "RemoveContainer" containerID="f4116d2eb6b57695f675f158238de954bbbad4d349b247b37d442f546e5bb49d"
Nov 03 08:13:23.472177 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:23.472154    5070 scope.go:122] "RemoveContainer" containerID="aa46d4b6d698a4e7b7cc4a5ad699e84723f812e4ae848bb11fbbd9f6d866d3a8"
Nov 03 08:13:23.482519 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:23.482500    5070 scope.go:122] "RemoveContainer" containerID="de26cf873275581845a1390e58363d09325fbc458ed82008aca6490eba57e0c4"
Nov 03 08:13:23.492199 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:23.492183    5070 scope.go:122] "RemoveContainer" containerID="a2d5421c7810e4ae75b249e62a1a6fc0a56ce16723cb1ec69a4d05bdb3ca2b36"
Nov 03 08:13:23.531112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:23.531096    5070 scope.go:122] "RemoveContainer" containerID="c8c3c7e1e424b062b9af045390c62e3017143b418d3d4659a5323c9fc992d70c"
Nov 03 08:13:23.541053 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:23.541030    5070 scope.go:122] "RemoveContainer" containerID="44d3dd8c9b947a32810f61647905a1f5cbf21ef1fbb2f1236cb251e7daaee94c"
Nov 03 08:13:23.551387 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:23.551372    5070 scope.go:122] "RemoveContainer" containerID="8d8115bb7c8959d8ef3c3ac545d5d8542e4797b275f0a0314e1dbdd33791404b"
Nov 03 08:13:23.561070 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:23.561055    5070 scope.go:122] "RemoveContainer" containerID="4aade9f7ab74300bd5fce8c071ff4879b112e7fe797b90fc23ecc9e382aa808c"
Nov 03 08:13:23.574083 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:23.574064    5070 scope.go:122] "RemoveContainer" containerID="321f21edfafd30811b4ae4ba6d728bef69ba1a570d2ce16a1d16302bff3d8d9a"
Nov 03 08:13:23.583741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:23.583724    5070 scope.go:122] "RemoveContainer" containerID="b5c62182f8f3b9d00b68bfc68b95b9e27e66086710dba7db67496ffd04fc19cc"
Nov 03 08:13:24.034437 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:24.034328    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw"]
Nov 03 08:13:24.040109 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:24.040088    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7e28100-v9xvw"]
Nov 03 08:13:26.033915 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:26.033882    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c7bcaf9-f7f9-429e-821c-ade13425d122" path="/var/lib/kubelet/pods/7c7bcaf9-f7f9-429e-821c-ade13425d122/volumes"
Nov 03 08:13:26.613953 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:26.613832    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b"]
Nov 03 08:13:26.618396 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:26.618369    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2a44ba-ltg6b"]
Nov 03 08:13:28.028368 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:28.028337    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eec2b89-b438-4e5b-b4fa-ac92fcc0d034" path="/var/lib/kubelet/pods/4eec2b89-b438-4e5b-b4fa-ac92fcc0d034/volumes"
Nov 03 08:13:29.038673 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:29.038636    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt"]
Nov 03 08:13:29.044210 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:29.044186    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-891bfe36-c2mjt"]
Nov 03 08:13:30.028341 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:30.028301    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c7858ef-04ee-4155-a69b-b3c3a11e4ea7" path="/var/lib/kubelet/pods/9c7858ef-04ee-4155-a69b-b3c3a11e4ea7/volumes"
Nov 03 08:13:34.513977 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:34.513893    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"]
Nov 03 08:13:34.514471 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:34.514179    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng" podUID="87e84029-163b-4605-8c14-8da36b5c6cf5" containerName="istio-proxy" containerID="cri-o://1570717c24ba0cf1971c9cf77416c998916652eff83a3ed7300b8cf95145c14a" gracePeriod=30
Nov 03 08:13:35.073213 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.073178    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt"]
Nov 03 08:13:35.074430 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.074383    5070 cpu_manager.go:414] "containerMap: removing container" podUID="a5f2321b-ee05-4de7-a27a-0e49d3102330" containerName="revoke-keys"
Nov 03 08:13:35.079837 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.079808    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f2321b-ee05-4de7-a27a-0e49d3102330" containerName="revoke-keys"
Nov 03 08:13:35.080062 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.080046    5070 cpu_manager.go:414] "containerMap: removing container" podUID="5d3ca38f-b7bc-431d-a1f9-aa69afe33648" containerName="revoke-keys"
Nov 03 08:13:35.080169 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.080155    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3ca38f-b7bc-431d-a1f9-aa69afe33648" containerName="revoke-keys"
Nov 03 08:13:35.080509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.080493    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d3ca38f-b7bc-431d-a1f9-aa69afe33648" containerName="revoke-keys"
Nov 03 08:13:35.080641 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.080630    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f2321b-ee05-4de7-a27a-0e49d3102330" containerName="revoke-keys"
Nov 03 08:13:35.099468 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.091103    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt"]
Nov 03 08:13:35.099468 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.091223    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt"
Nov 03 08:13:35.108031 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.105719    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:13:35.109483 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.109460    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:13:35.168800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.167598    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:13:35.168800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.167626    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:13:35.253304 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.253271    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f00ee5-7b50-4653-9405-95d306eb87cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dfb883-m92kt\" (UID: \"54f00ee5-7b50-4653-9405-95d306eb87cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt"
Nov 03 08:13:35.253479 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.253463    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch5g6\" (UniqueName: \"kubernetes.io/projected/54f00ee5-7b50-4653-9405-95d306eb87cf-kube-api-access-ch5g6\") pod \"maas-api-revoke-keys-e2e-api-b-dfb883-m92kt\" (UID: \"54f00ee5-7b50-4653-9405-95d306eb87cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt"
Nov 03 08:13:35.253639 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.253624    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f00ee5-7b50-4653-9405-95d306eb87cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dfb883-m92kt\" (UID: \"54f00ee5-7b50-4653-9405-95d306eb87cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt"
Nov 03 08:13:35.354668 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.354630    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f00ee5-7b50-4653-9405-95d306eb87cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dfb883-m92kt\" (UID: \"54f00ee5-7b50-4653-9405-95d306eb87cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt"
Nov 03 08:13:35.354886 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.354847    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch5g6\" (UniqueName: \"kubernetes.io/projected/54f00ee5-7b50-4653-9405-95d306eb87cf-kube-api-access-ch5g6\") pod \"maas-api-revoke-keys-e2e-api-b-dfb883-m92kt\" (UID: \"54f00ee5-7b50-4653-9405-95d306eb87cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt"
Nov 03 08:13:35.354994 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.354977    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f00ee5-7b50-4653-9405-95d306eb87cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dfb883-m92kt\" (UID: \"54f00ee5-7b50-4653-9405-95d306eb87cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt"
Nov 03 08:13:35.355407 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.355377    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f00ee5-7b50-4653-9405-95d306eb87cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dfb883-m92kt\" (UID: \"54f00ee5-7b50-4653-9405-95d306eb87cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt"
Nov 03 08:13:35.358477 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.358454    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f00ee5-7b50-4653-9405-95d306eb87cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dfb883-m92kt\" (UID: \"54f00ee5-7b50-4653-9405-95d306eb87cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt"
Nov 03 08:13:35.362526 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.362502    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch5g6\" (UniqueName: \"kubernetes.io/projected/54f00ee5-7b50-4653-9405-95d306eb87cf-kube-api-access-ch5g6\") pod \"maas-api-revoke-keys-e2e-api-b-dfb883-m92kt\" (UID: \"54f00ee5-7b50-4653-9405-95d306eb87cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt"
Nov 03 08:13:35.429644 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.429613    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt"
Nov 03 08:13:35.757443 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:13:35.757415    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f00ee5_7b50_4653_9405_95d306eb87cf.slice/crio-de93996d47b1666ba8bbe77e30117ff6568c1ec504785b44fbca600486161525 WatchSource:0}: Error finding container de93996d47b1666ba8bbe77e30117ff6568c1ec504785b44fbca600486161525: Status 404 returned error can't find the container with id de93996d47b1666ba8bbe77e30117ff6568c1ec504785b44fbca600486161525
Nov 03 08:13:35.757801 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:35.757711    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt"]
Nov 03 08:13:36.206417 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:36.206385    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt" event={"ID":"54f00ee5-7b50-4653-9405-95d306eb87cf","Type":"ContainerStarted","Data":"23005b5dbc4e80e62b6ae5e46757a40b93769b4e96bf189924ca7104dc752464"}
Nov 03 08:13:36.206573 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:36.206441    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt" event={"ID":"54f00ee5-7b50-4653-9405-95d306eb87cf","Type":"ContainerStarted","Data":"de93996d47b1666ba8bbe77e30117ff6568c1ec504785b44fbca600486161525"}
Nov 03 08:13:36.226323 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:36.226280    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt" podStartSLOduration=1.226265786 podStartE2EDuration="1.226265786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:13:36.222866819 +0000 UTC m=+1214.337330750" watchObservedRunningTime="2026-11-03 08:13:36.226265786 +0000 UTC m=+1214.340729714"
Nov 03 08:13:37.217057 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:37.217025    5070 generic.go:356] "Generic (PLEG): container finished" podID="54f00ee5-7b50-4653-9405-95d306eb87cf" containerID="23005b5dbc4e80e62b6ae5e46757a40b93769b4e96bf189924ca7104dc752464" exitCode=0
Nov 03 08:13:37.217506 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:37.217102    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt" event={"ID":"54f00ee5-7b50-4653-9405-95d306eb87cf","Type":"ContainerDied","Data":"23005b5dbc4e80e62b6ae5e46757a40b93769b4e96bf189924ca7104dc752464"}
Nov 03 08:13:38.460336 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:38.460307    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt"
Nov 03 08:13:38.621537 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:38.621488    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54f00ee5-7b50-4653-9405-95d306eb87cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f00ee5-7b50-4653-9405-95d306eb87cf-maas-api-openshift-ca\") pod \"54f00ee5-7b50-4653-9405-95d306eb87cf\" (UID: \"54f00ee5-7b50-4653-9405-95d306eb87cf\") "
Nov 03 08:13:38.622685 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:38.622650    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54f00ee5-7b50-4653-9405-95d306eb87cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f00ee5-7b50-4653-9405-95d306eb87cf-maas-api-xks-ca\") pod \"54f00ee5-7b50-4653-9405-95d306eb87cf\" (UID: \"54f00ee5-7b50-4653-9405-95d306eb87cf\") "
Nov 03 08:13:38.623502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:38.623478    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f00ee5-7b50-4653-9405-95d306eb87cf-kube-api-access-ch5g6\" (UniqueName: \"kubernetes.io/projected/54f00ee5-7b50-4653-9405-95d306eb87cf-kube-api-access-ch5g6\") pod \"54f00ee5-7b50-4653-9405-95d306eb87cf\" (UID: \"54f00ee5-7b50-4653-9405-95d306eb87cf\") "
Nov 03 08:13:38.629311 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:38.622344    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f00ee5-7b50-4653-9405-95d306eb87cf-maas-api-openshift-ca" pod "54f00ee5-7b50-4653-9405-95d306eb87cf" (UID: "54f00ee5-7b50-4653-9405-95d306eb87cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:13:38.648392 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:38.645440    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54f00ee5-7b50-4653-9405-95d306eb87cf-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:38.681391 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:38.681279    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54f00ee5-7b50-4653-9405-95d306eb87cf-maas-api-xks-ca" pod "54f00ee5-7b50-4653-9405-95d306eb87cf" (UID: "54f00ee5-7b50-4653-9405-95d306eb87cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:13:38.728239 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:38.728194    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f00ee5-7b50-4653-9405-95d306eb87cf-kube-api-access-ch5g6" pod "54f00ee5-7b50-4653-9405-95d306eb87cf" (UID: "54f00ee5-7b50-4653-9405-95d306eb87cf"). InnerVolumeSpecName "kube-api-access-ch5g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:13:38.749216 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:38.749173    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54f00ee5-7b50-4653-9405-95d306eb87cf-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:38.749216 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:38.749211    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch5g6\" (UniqueName: \"kubernetes.io/projected/54f00ee5-7b50-4653-9405-95d306eb87cf-kube-api-access-ch5g6\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:39.336816 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:39.332075    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt" event={"ID":"54f00ee5-7b50-4653-9405-95d306eb87cf","Type":"ContainerDied","Data":"de93996d47b1666ba8bbe77e30117ff6568c1ec504785b44fbca600486161525"}
Nov 03 08:13:39.336816 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:39.332156    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de93996d47b1666ba8bbe77e30117ff6568c1ec504785b44fbca600486161525"
Nov 03 08:13:39.336816 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:39.332254    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt"
Nov 03 08:13:40.137270 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.137245    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:13:40.200094 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.200063    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-workload-socket\") pod \"87e84029-163b-4605-8c14-8da36b5c6cf5\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") "
Nov 03 08:13:40.200296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.200202    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-envoy\") pod \"87e84029-163b-4605-8c14-8da36b5c6cf5\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") "
Nov 03 08:13:40.200296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.200246    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-workload-certs\") pod \"87e84029-163b-4605-8c14-8da36b5c6cf5\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") "
Nov 03 08:13:40.200296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.200287    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e84029-163b-4605-8c14-8da36b5c6cf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e84029-163b-4605-8c14-8da36b5c6cf5-istiod-ca-cert\") pod \"87e84029-163b-4605-8c14-8da36b5c6cf5\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") "
Nov 03 08:13:40.200471 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.200366    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e84029-163b-4605-8c14-8da36b5c6cf5-kube-api-access-9dfgb\" (UniqueName: \"kubernetes.io/projected/87e84029-163b-4605-8c14-8da36b5c6cf5-kube-api-access-9dfgb\") pod \"87e84029-163b-4605-8c14-8da36b5c6cf5\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") "
Nov 03 08:13:40.200471 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.200406    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-workload-socket" pod "87e84029-163b-4605-8c14-8da36b5c6cf5" (UID: "87e84029-163b-4605-8c14-8da36b5c6cf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:13:40.200471 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.200464    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-token\" (UniqueName: \"kubernetes.io/projected/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-token\") pod \"87e84029-163b-4605-8c14-8da36b5c6cf5\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") "
Nov 03 08:13:40.200637 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.200541    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-data\") pod \"87e84029-163b-4605-8c14-8da36b5c6cf5\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") "
Nov 03 08:13:40.200637 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.200584    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-podinfo\") pod \"87e84029-163b-4605-8c14-8da36b5c6cf5\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") "
Nov 03 08:13:40.200743 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.200678    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-credential-socket\") pod \"87e84029-163b-4605-8c14-8da36b5c6cf5\" (UID: \"87e84029-163b-4605-8c14-8da36b5c6cf5\") "
Nov 03 08:13:40.200866 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.200833    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e84029-163b-4605-8c14-8da36b5c6cf5-istiod-ca-cert" pod "87e84029-163b-4605-8c14-8da36b5c6cf5" (UID: "87e84029-163b-4605-8c14-8da36b5c6cf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:13:40.201006 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.200981    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-workload-certs" pod "87e84029-163b-4605-8c14-8da36b5c6cf5" (UID: "87e84029-163b-4605-8c14-8da36b5c6cf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:13:40.201402 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.201375    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:40.201402 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.201404    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:40.201566 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.201414    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87e84029-163b-4605-8c14-8da36b5c6cf5-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:40.201628 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.201608    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-data" pod "87e84029-163b-4605-8c14-8da36b5c6cf5" (UID: "87e84029-163b-4605-8c14-8da36b5c6cf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:13:40.201628 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.201618    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-credential-socket" pod "87e84029-163b-4605-8c14-8da36b5c6cf5" (UID: "87e84029-163b-4605-8c14-8da36b5c6cf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:13:40.203046 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.203019    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-envoy" pod "87e84029-163b-4605-8c14-8da36b5c6cf5" (UID: "87e84029-163b-4605-8c14-8da36b5c6cf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:13:40.203733 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.203704    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e84029-163b-4605-8c14-8da36b5c6cf5-kube-api-access-9dfgb" pod "87e84029-163b-4605-8c14-8da36b5c6cf5" (UID: "87e84029-163b-4605-8c14-8da36b5c6cf5"). InnerVolumeSpecName "kube-api-access-9dfgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:13:40.204119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.204082    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-podinfo" pod "87e84029-163b-4605-8c14-8da36b5c6cf5" (UID: "87e84029-163b-4605-8c14-8da36b5c6cf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:13:40.204678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.204657    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-token" pod "87e84029-163b-4605-8c14-8da36b5c6cf5" (UID: "87e84029-163b-4605-8c14-8da36b5c6cf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:13:40.302812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.302745    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dfgb\" (UniqueName: \"kubernetes.io/projected/87e84029-163b-4605-8c14-8da36b5c6cf5-kube-api-access-9dfgb\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:40.302812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.302812    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:40.303008 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.302822    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:40.303008 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.302832    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:40.303008 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.302841    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:40.303008 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.302850    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87e84029-163b-4605-8c14-8da36b5c6cf5-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:13:40.344055 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.344018    5070 generic.go:356] "Generic (PLEG): container finished" podID="87e84029-163b-4605-8c14-8da36b5c6cf5" containerID="1570717c24ba0cf1971c9cf77416c998916652eff83a3ed7300b8cf95145c14a" exitCode=0
Nov 03 08:13:40.344201 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.344061    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"
Nov 03 08:13:40.344201 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.344094    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng" event={"ID":"87e84029-163b-4605-8c14-8da36b5c6cf5","Type":"ContainerDied","Data":"1570717c24ba0cf1971c9cf77416c998916652eff83a3ed7300b8cf95145c14a"}
Nov 03 08:13:40.344201 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.344148    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng" event={"ID":"87e84029-163b-4605-8c14-8da36b5c6cf5","Type":"ContainerDied","Data":"a0154d76e02edf72b9a514181c68daec296cfab7353152093b76e2b98bcc915e"}
Nov 03 08:13:40.344201 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.344178    5070 scope.go:122] "RemoveContainer" containerID="1570717c24ba0cf1971c9cf77416c998916652eff83a3ed7300b8cf95145c14a"
Nov 03 08:13:40.360602 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.360580    5070 scope.go:122] "RemoveContainer" containerID="805447d0005e492d53887463bb38cc18474ffbe1a7a32d179f76767976b14712"
Nov 03 08:13:40.372117 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.372096    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"]
Nov 03 08:13:40.372926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.372908    5070 scope.go:122] "RemoveContainer" containerID="1570717c24ba0cf1971c9cf77416c998916652eff83a3ed7300b8cf95145c14a"
Nov 03 08:13:40.373689 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:13:40.373258    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1570717c24ba0cf1971c9cf77416c998916652eff83a3ed7300b8cf95145c14a\": container with ID starting with 1570717c24ba0cf1971c9cf77416c998916652eff83a3ed7300b8cf95145c14a not found: ID does not exist" containerID="1570717c24ba0cf1971c9cf77416c998916652eff83a3ed7300b8cf95145c14a"
Nov 03 08:13:40.373689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.373295    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1570717c24ba0cf1971c9cf77416c998916652eff83a3ed7300b8cf95145c14a"} err="failed to get container status \"1570717c24ba0cf1971c9cf77416c998916652eff83a3ed7300b8cf95145c14a\": rpc error: code = NotFound desc = could not find container \"1570717c24ba0cf1971c9cf77416c998916652eff83a3ed7300b8cf95145c14a\": container with ID starting with 1570717c24ba0cf1971c9cf77416c998916652eff83a3ed7300b8cf95145c14a not found: ID does not exist"
Nov 03 08:13:40.373689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.373321    5070 scope.go:122] "RemoveContainer" containerID="805447d0005e492d53887463bb38cc18474ffbe1a7a32d179f76767976b14712"
Nov 03 08:13:40.373988 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:13:40.373965    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"805447d0005e492d53887463bb38cc18474ffbe1a7a32d179f76767976b14712\": container with ID starting with 805447d0005e492d53887463bb38cc18474ffbe1a7a32d179f76767976b14712 not found: ID does not exist" containerID="805447d0005e492d53887463bb38cc18474ffbe1a7a32d179f76767976b14712"
Nov 03 08:13:40.374069 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.373995    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"805447d0005e492d53887463bb38cc18474ffbe1a7a32d179f76767976b14712"} err="failed to get container status \"805447d0005e492d53887463bb38cc18474ffbe1a7a32d179f76767976b14712\": rpc error: code = NotFound desc = could not find container \"805447d0005e492d53887463bb38cc18474ffbe1a7a32d179f76767976b14712\": container with ID starting with 805447d0005e492d53887463bb38cc18474ffbe1a7a32d179f76767976b14712 not found: ID does not exist"
Nov 03 08:13:40.380336 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:40.380313    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2a44ba-openshift-default-5c874f864f-sfqng"]
Nov 03 08:13:42.052371 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:42.052340    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e84029-163b-4605-8c14-8da36b5c6cf5" path="/var/lib/kubelet/pods/87e84029-163b-4605-8c14-8da36b5c6cf5/volumes"
Nov 03 08:13:52.054080 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:13:52.054018    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:13:54.927891 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:54.927854    5070 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:13:54.928317 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:54.927918    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr" podUID="58a77803-4112-45fe-bf9b-5ca069a5441f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:13:58.048556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.048516    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d"]
Nov 03 08:13:58.049346 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.049329    5070 cpu_manager.go:414] "containerMap: removing container" podUID="54f00ee5-7b50-4653-9405-95d306eb87cf" containerName="revoke-keys"
Nov 03 08:13:58.049391 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.049348    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f00ee5-7b50-4653-9405-95d306eb87cf" containerName="revoke-keys"
Nov 03 08:13:58.049431 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.049405    5070 cpu_manager.go:414] "containerMap: removing container" podUID="87e84029-163b-4605-8c14-8da36b5c6cf5" containerName="istio-proxy"
Nov 03 08:13:58.049431 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.049411    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e84029-163b-4605-8c14-8da36b5c6cf5" containerName="istio-proxy"
Nov 03 08:13:58.049498 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.049432    5070 cpu_manager.go:414] "containerMap: removing container" podUID="87e84029-163b-4605-8c14-8da36b5c6cf5" containerName="istio-proxy"
Nov 03 08:13:58.049498 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.049438    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e84029-163b-4605-8c14-8da36b5c6cf5" containerName="istio-proxy"
Nov 03 08:13:58.049567 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.049557    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e84029-163b-4605-8c14-8da36b5c6cf5" containerName="istio-proxy"
Nov 03 08:13:58.049601 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.049574    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e84029-163b-4605-8c14-8da36b5c6cf5" containerName="istio-proxy"
Nov 03 08:13:58.049601 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.049582    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f00ee5-7b50-4653-9405-95d306eb87cf" containerName="revoke-keys"
Nov 03 08:13:58.245328 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.245296    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d"]
Nov 03 08:13:58.245540 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.245407    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d"
Nov 03 08:13:58.254145 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.254122    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:13:58.254266 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.254195    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:13:58.254414 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.254398    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:13:58.254525 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.254506    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:13:58.347244 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.347169    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wf22\" (UniqueName: \"kubernetes.io/projected/43306943-c1e8-4bb2-be03-c540de81d41d-kube-api-access-9wf22\") pod \"maas-api-revoke-keys-e2e-mt-cb887217-g6x8d\" (UID: \"43306943-c1e8-4bb2-be03-c540de81d41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d"
Nov 03 08:13:58.347441 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.347420    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43306943-c1e8-4bb2-be03-c540de81d41d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb887217-g6x8d\" (UID: \"43306943-c1e8-4bb2-be03-c540de81d41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d"
Nov 03 08:13:58.347583 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.347482    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43306943-c1e8-4bb2-be03-c540de81d41d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb887217-g6x8d\" (UID: \"43306943-c1e8-4bb2-be03-c540de81d41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d"
Nov 03 08:13:58.449146 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.449116    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43306943-c1e8-4bb2-be03-c540de81d41d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb887217-g6x8d\" (UID: \"43306943-c1e8-4bb2-be03-c540de81d41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d"
Nov 03 08:13:58.449319 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.449192    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wf22\" (UniqueName: \"kubernetes.io/projected/43306943-c1e8-4bb2-be03-c540de81d41d-kube-api-access-9wf22\") pod \"maas-api-revoke-keys-e2e-mt-cb887217-g6x8d\" (UID: \"43306943-c1e8-4bb2-be03-c540de81d41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d"
Nov 03 08:13:58.449497 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.449474    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43306943-c1e8-4bb2-be03-c540de81d41d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb887217-g6x8d\" (UID: \"43306943-c1e8-4bb2-be03-c540de81d41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d"
Nov 03 08:13:58.450140 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.450118    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43306943-c1e8-4bb2-be03-c540de81d41d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb887217-g6x8d\" (UID: \"43306943-c1e8-4bb2-be03-c540de81d41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d"
Nov 03 08:13:58.519926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.519880    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43306943-c1e8-4bb2-be03-c540de81d41d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb887217-g6x8d\" (UID: \"43306943-c1e8-4bb2-be03-c540de81d41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d"
Nov 03 08:13:58.522129 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.522090    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wf22\" (UniqueName: \"kubernetes.io/projected/43306943-c1e8-4bb2-be03-c540de81d41d-kube-api-access-9wf22\") pod \"maas-api-revoke-keys-e2e-mt-cb887217-g6x8d\" (UID: \"43306943-c1e8-4bb2-be03-c540de81d41d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d"
Nov 03 08:13:58.598831 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.598417    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t"]
Nov 03 08:13:58.668944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.664888    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t"]
Nov 03 08:13:58.668944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.665009    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t"
Nov 03 08:13:58.690143 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.690022    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d"
Nov 03 08:13:58.776538 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.776496    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3866ec89-7dfb-43db-acf0-ea0c3069c989-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t\" (UID: \"3866ec89-7dfb-43db-acf0-ea0c3069c989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t"
Nov 03 08:13:58.776728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.776671    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3866ec89-7dfb-43db-acf0-ea0c3069c989-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t\" (UID: \"3866ec89-7dfb-43db-acf0-ea0c3069c989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t"
Nov 03 08:13:58.776817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.776751    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f79g\" (UniqueName: \"kubernetes.io/projected/3866ec89-7dfb-43db-acf0-ea0c3069c989-kube-api-access-5f79g\") pod \"maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t\" (UID: \"3866ec89-7dfb-43db-acf0-ea0c3069c989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t"
Nov 03 08:13:58.878829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.878796    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3866ec89-7dfb-43db-acf0-ea0c3069c989-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t\" (UID: \"3866ec89-7dfb-43db-acf0-ea0c3069c989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t"
Nov 03 08:13:58.878999 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.878951    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3866ec89-7dfb-43db-acf0-ea0c3069c989-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t\" (UID: \"3866ec89-7dfb-43db-acf0-ea0c3069c989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t"
Nov 03 08:13:58.879063 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.879033    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f79g\" (UniqueName: \"kubernetes.io/projected/3866ec89-7dfb-43db-acf0-ea0c3069c989-kube-api-access-5f79g\") pod \"maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t\" (UID: \"3866ec89-7dfb-43db-acf0-ea0c3069c989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t"
Nov 03 08:13:58.879604 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.879578    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3866ec89-7dfb-43db-acf0-ea0c3069c989-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t\" (UID: \"3866ec89-7dfb-43db-acf0-ea0c3069c989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t"
Nov 03 08:13:58.889716 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.889690    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f79g\" (UniqueName: \"kubernetes.io/projected/3866ec89-7dfb-43db-acf0-ea0c3069c989-kube-api-access-5f79g\") pod \"maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t\" (UID: \"3866ec89-7dfb-43db-acf0-ea0c3069c989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t"
Nov 03 08:13:58.889859 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:58.889744    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3866ec89-7dfb-43db-acf0-ea0c3069c989-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t\" (UID: \"3866ec89-7dfb-43db-acf0-ea0c3069c989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t"
Nov 03 08:13:59.105637 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:59.105605    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t"
Nov 03 08:13:59.863284 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:59.863250    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"]
Nov 03 08:13:59.990783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:13:59.988988    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.002789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.000828    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-228695-openshift-default-dockercfg-9dgpq\""
Nov 03 08:14:00.055668 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.047911    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-envoy\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.055668 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.047964    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63d796bd-8897-4067-b1b4-e39aecd06b88-istiod-ca-cert\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.055668 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.048000    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-podinfo\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.055668 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.048064    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sfwl\" (UniqueName: \"kubernetes.io/projected/63d796bd-8897-4067-b1b4-e39aecd06b88-kube-api-access-9sfwl\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.055668 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.048166    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-data\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.055668 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.048253    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-workload-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.055668 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.048305    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-token\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.055668 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.048342    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-workload-certs\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.055668 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.048460    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-credential-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.174783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.173960    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-token\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.174783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.174026    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-workload-certs\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.174783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.174340    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-credential-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.174783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.174440    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-envoy\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.174783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.174472    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63d796bd-8897-4067-b1b4-e39aecd06b88-istiod-ca-cert\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.174783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.174506    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-podinfo\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.174783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.174588    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sfwl\" (UniqueName: \"kubernetes.io/projected/63d796bd-8897-4067-b1b4-e39aecd06b88-kube-api-access-9sfwl\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.192234 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.174790    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-data\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.192234 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.175772    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-workload-certs\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.192234 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.175922    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-workload-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.192234 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.176318    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-workload-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.192234 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.176553    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-data\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.192496 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.192457    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63d796bd-8897-4067-b1b4-e39aecd06b88-istiod-ca-cert\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.193110 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.192715    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-credential-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.285800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.257169    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-envoy\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.285800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.257603    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-token\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.299624 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.298465    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sfwl\" (UniqueName: \"kubernetes.io/projected/63d796bd-8897-4067-b1b4-e39aecd06b88-kube-api-access-9sfwl\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.300121 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.300025    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-podinfo\") pod \"e2e-api-b-228695-openshift-default-547df686cf-n5bqg\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:00.325789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.311287    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"]
Nov 03 08:14:00.521325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:00.521233    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:01.896820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:01.679975    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d"]
Nov 03 08:14:01.896820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:01.717062    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"]
Nov 03 08:14:01.896820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:01.721916    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t"]
Nov 03 08:14:02.107296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:02.050275    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:14:02.107296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:02.050382    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:14:02.107296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:02.050422    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:14:02.851676 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:02.851646    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d" event={"ID":"43306943-c1e8-4bb2-be03-c540de81d41d","Type":"ContainerStarted","Data":"32572cdd126b8ed2a355fc652f8f7e9fed7619efd04600a7e77a9e4a8548f7ca"}
Nov 03 08:14:02.853053 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:02.853030    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t" event={"ID":"3866ec89-7dfb-43db-acf0-ea0c3069c989","Type":"ContainerStarted","Data":"8dafd21d1847cbb67ff8bcd73f26da43e28a9f01200616bc152d9e15abffac33"}
Nov 03 08:14:02.890988 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:02.854324    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg" event={"ID":"63d796bd-8897-4067-b1b4-e39aecd06b88","Type":"ContainerStarted","Data":"faae2c263262f291bd40eb3ba6bebb8859cb3d5d3536aaa53324d50f05071dae"}
Nov 03 08:14:04.908245 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:04.908201    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t" event={"ID":"3866ec89-7dfb-43db-acf0-ea0c3069c989","Type":"ContainerStarted","Data":"04e1e47da8e5b4144e964a513167699f2b9279b527163f28bde1351a49c1cffb"}
Nov 03 08:14:04.910148 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:04.910117    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg" event={"ID":"63d796bd-8897-4067-b1b4-e39aecd06b88","Type":"ContainerStarted","Data":"b11717b193b9d6894c25850a626a8a1063989c6c412a8f4f6606a2e54788de64"}
Nov 03 08:14:04.912053 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:04.912027    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d" event={"ID":"43306943-c1e8-4bb2-be03-c540de81d41d","Type":"ContainerStarted","Data":"349234bc369fa4aea009b05985d7d0d87fe929a99f81a9eafe4acf522c95a9b6"}
Nov 03 08:14:04.928295 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:04.928235    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t" podStartSLOduration=6.928210004 podStartE2EDuration="6.928210004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:04.921738509 +0000 UTC m=+1243.036202437" watchObservedRunningTime="2026-11-03 08:14:04.928210004 +0000 UTC m=+1243.042673936"
Nov 03 08:14:04.939281 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:04.939240    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d" podStartSLOduration=6.939227352 podStartE2EDuration="6.939227352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:04.935000364 +0000 UTC m=+1243.049464295" watchObservedRunningTime="2026-11-03 08:14:04.939227352 +0000 UTC m=+1243.053691282"
Nov 03 08:14:04.955130 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:04.955092    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg" podStartSLOduration=5.95508135 podStartE2EDuration="5.95508135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:04.950658926 +0000 UTC m=+1243.065122857" watchObservedRunningTime="2026-11-03 08:14:04.95508135 +0000 UTC m=+1243.069545338"
Nov 03 08:14:05.521656 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:05.521624    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:05.525227 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:05.525197    5070 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-n5bqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Nov 03 08:14:05.525366 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:05.525256    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg" podUID="63d796bd-8897-4067-b1b4-e39aecd06b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Nov 03 08:14:05.927869 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:05.927836    5070 generic.go:356] "Generic (PLEG): container finished" podID="43306943-c1e8-4bb2-be03-c540de81d41d" containerID="349234bc369fa4aea009b05985d7d0d87fe929a99f81a9eafe4acf522c95a9b6" exitCode=0
Nov 03 08:14:05.928443 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:05.928002    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d" event={"ID":"43306943-c1e8-4bb2-be03-c540de81d41d","Type":"ContainerDied","Data":"349234bc369fa4aea009b05985d7d0d87fe929a99f81a9eafe4acf522c95a9b6"}
Nov 03 08:14:05.930263 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:05.930225    5070 generic.go:356] "Generic (PLEG): container finished" podID="3866ec89-7dfb-43db-acf0-ea0c3069c989" containerID="04e1e47da8e5b4144e964a513167699f2b9279b527163f28bde1351a49c1cffb" exitCode=0
Nov 03 08:14:05.930419 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:05.930256    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t" event={"ID":"3866ec89-7dfb-43db-acf0-ea0c3069c989","Type":"ContainerDied","Data":"04e1e47da8e5b4144e964a513167699f2b9279b527163f28bde1351a49c1cffb"}
Nov 03 08:14:06.531328 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:06.531294    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:06.677124 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:06.677088    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt"]
Nov 03 08:14:06.682533 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:06.682507    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dfb883-m92kt"]
Nov 03 08:14:06.977082 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:06.942601    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:06.977082 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:06.943731    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:14:07.541623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.480412    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t"
Nov 03 08:14:07.541623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.496375    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d"
Nov 03 08:14:07.761413 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.579539    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3866ec89-7dfb-43db-acf0-ea0c3069c989-kube-api-access-5f79g\" (UniqueName: \"kubernetes.io/projected/3866ec89-7dfb-43db-acf0-ea0c3069c989-kube-api-access-5f79g\") pod \"3866ec89-7dfb-43db-acf0-ea0c3069c989\" (UID: \"3866ec89-7dfb-43db-acf0-ea0c3069c989\") "
Nov 03 08:14:07.761413 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.579583    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43306943-c1e8-4bb2-be03-c540de81d41d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43306943-c1e8-4bb2-be03-c540de81d41d-maas-api-xks-ca\") pod \"43306943-c1e8-4bb2-be03-c540de81d41d\" (UID: \"43306943-c1e8-4bb2-be03-c540de81d41d\") "
Nov 03 08:14:07.761413 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.579749    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3866ec89-7dfb-43db-acf0-ea0c3069c989-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3866ec89-7dfb-43db-acf0-ea0c3069c989-maas-api-xks-ca\") pod \"3866ec89-7dfb-43db-acf0-ea0c3069c989\" (UID: \"3866ec89-7dfb-43db-acf0-ea0c3069c989\") "
Nov 03 08:14:07.761413 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.579855    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43306943-c1e8-4bb2-be03-c540de81d41d-kube-api-access-9wf22\" (UniqueName: \"kubernetes.io/projected/43306943-c1e8-4bb2-be03-c540de81d41d-kube-api-access-9wf22\") pod \"43306943-c1e8-4bb2-be03-c540de81d41d\" (UID: \"43306943-c1e8-4bb2-be03-c540de81d41d\") "
Nov 03 08:14:07.761413 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.579916    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3866ec89-7dfb-43db-acf0-ea0c3069c989-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3866ec89-7dfb-43db-acf0-ea0c3069c989-maas-api-openshift-ca\") pod \"3866ec89-7dfb-43db-acf0-ea0c3069c989\" (UID: \"3866ec89-7dfb-43db-acf0-ea0c3069c989\") "
Nov 03 08:14:07.761413 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.580045    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43306943-c1e8-4bb2-be03-c540de81d41d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43306943-c1e8-4bb2-be03-c540de81d41d-maas-api-openshift-ca\") pod \"43306943-c1e8-4bb2-be03-c540de81d41d\" (UID: \"43306943-c1e8-4bb2-be03-c540de81d41d\") "
Nov 03 08:14:07.761413 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.580390    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3866ec89-7dfb-43db-acf0-ea0c3069c989-maas-api-openshift-ca" pod "3866ec89-7dfb-43db-acf0-ea0c3069c989" (UID: "3866ec89-7dfb-43db-acf0-ea0c3069c989"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:14:07.761413 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.580446    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43306943-c1e8-4bb2-be03-c540de81d41d-maas-api-openshift-ca" pod "43306943-c1e8-4bb2-be03-c540de81d41d" (UID: "43306943-c1e8-4bb2-be03-c540de81d41d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:14:07.761413 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.580979    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3866ec89-7dfb-43db-acf0-ea0c3069c989-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:07.761413 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.581005    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43306943-c1e8-4bb2-be03-c540de81d41d-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:07.929989 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.788808    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43306943-c1e8-4bb2-be03-c540de81d41d-maas-api-xks-ca" pod "43306943-c1e8-4bb2-be03-c540de81d41d" (UID: "43306943-c1e8-4bb2-be03-c540de81d41d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:14:07.929989 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.789032    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3866ec89-7dfb-43db-acf0-ea0c3069c989-kube-api-access-5f79g" pod "3866ec89-7dfb-43db-acf0-ea0c3069c989" (UID: "3866ec89-7dfb-43db-acf0-ea0c3069c989"). InnerVolumeSpecName "kube-api-access-5f79g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:07.929989 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.803798    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3866ec89-7dfb-43db-acf0-ea0c3069c989-maas-api-xks-ca" pod "3866ec89-7dfb-43db-acf0-ea0c3069c989" (UID: "3866ec89-7dfb-43db-acf0-ea0c3069c989"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:14:07.929989 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.805411    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43306943-c1e8-4bb2-be03-c540de81d41d-kube-api-access-9wf22" pod "43306943-c1e8-4bb2-be03-c540de81d41d" (UID: "43306943-c1e8-4bb2-be03-c540de81d41d"). InnerVolumeSpecName "kube-api-access-9wf22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:07.929989 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.906588    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f79g\" (UniqueName: \"kubernetes.io/projected/3866ec89-7dfb-43db-acf0-ea0c3069c989-kube-api-access-5f79g\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:07.929989 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.906624    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43306943-c1e8-4bb2-be03-c540de81d41d-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:07.929989 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.906644    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3866ec89-7dfb-43db-acf0-ea0c3069c989-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:07.929989 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.906658    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wf22\" (UniqueName: \"kubernetes.io/projected/43306943-c1e8-4bb2-be03-c540de81d41d-kube-api-access-9wf22\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:08.013978 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.966801    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d"
Nov 03 08:14:08.013978 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.966831    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d" event={"ID":"43306943-c1e8-4bb2-be03-c540de81d41d","Type":"ContainerDied","Data":"32572cdd126b8ed2a355fc652f8f7e9fed7619efd04600a7e77a9e4a8548f7ca"}
Nov 03 08:14:08.013978 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.966894    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32572cdd126b8ed2a355fc652f8f7e9fed7619efd04600a7e77a9e4a8548f7ca"
Nov 03 08:14:08.013978 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.986552    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t" event={"ID":"3866ec89-7dfb-43db-acf0-ea0c3069c989","Type":"ContainerDied","Data":"8dafd21d1847cbb67ff8bcd73f26da43e28a9f01200616bc152d9e15abffac33"}
Nov 03 08:14:08.013978 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.986633    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dafd21d1847cbb67ff8bcd73f26da43e28a9f01200616bc152d9e15abffac33"
Nov 03 08:14:08.013978 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:07.986849    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t"
Nov 03 08:14:08.037309 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:08.029468    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54f00ee5-7b50-4653-9405-95d306eb87cf" path="/var/lib/kubelet/pods/54f00ee5-7b50-4653-9405-95d306eb87cf/volumes"
Nov 03 08:14:09.518540 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:09.518504    5070 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": read tcp 10.132.0.2:56578->10.132.0.167:15021: read: connection reset by peer" start-of-body=
Nov 03 08:14:09.519034 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:09.518595    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr" podUID="58a77803-4112-45fe-bf9b-5ca069a5441f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": read tcp 10.132.0.2:56578->10.132.0.167:15021: read: connection reset by peer"
Nov 03 08:14:10.011654 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:10.011623    5070 generic.go:356] "Generic (PLEG): container finished" podID="58a77803-4112-45fe-bf9b-5ca069a5441f" containerID="0df6c92f2ed4ae98ac879da0364e52048aa4c49e89ea406ac019138d1ae36d7a" exitCode=0
Nov 03 08:14:10.011855 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:10.011693    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr" event={"ID":"58a77803-4112-45fe-bf9b-5ca069a5441f","Type":"ContainerDied","Data":"0df6c92f2ed4ae98ac879da0364e52048aa4c49e89ea406ac019138d1ae36d7a"}
Nov 03 08:14:10.013826 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:10.013807    5070 scope.go:122] "RemoveContainer" containerID="0df6c92f2ed4ae98ac879da0364e52048aa4c49e89ea406ac019138d1ae36d7a"
Nov 03 08:14:10.015604 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:10.015570    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:14:10.015704 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:10.015643    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:14:10.015704 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:10.015676    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:14:11.100817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:11.100639    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr" event={"ID":"58a77803-4112-45fe-bf9b-5ca069a5441f","Type":"ContainerStarted","Data":"71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997"}
Nov 03 08:14:11.965076 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:11.964161    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:14:11.965076 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:11.964299    5070 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 03 08:14:11.965076 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:11.964344    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr" podUID="58a77803-4112-45fe-bf9b-5ca069a5441f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 03 08:14:12.988779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:12.987899    5070 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 03 08:14:12.988779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:12.987974    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr" podUID="58a77803-4112-45fe-bf9b-5ca069a5441f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 03 08:14:13.976784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:13.974176    5070 patch_prober.go:29] interesting pod/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body=
Nov 03 08:14:13.976784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:13.974240    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr" podUID="58a77803-4112-45fe-bf9b-5ca069a5441f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused"
Nov 03 08:14:14.206669 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:14.206631    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"]
Nov 03 08:14:14.207244 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:14.206903    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr" podUID="58a77803-4112-45fe-bf9b-5ca069a5441f" containerName="istio-proxy" containerID="cri-o://71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997" gracePeriod=30
Nov 03 08:14:16.692845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:16.692809    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"]
Nov 03 08:14:16.693264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:16.693025    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5" podUID="2a7d0e21-8828-4ebd-a135-068d713fe575" containerName="istio-proxy" containerID="cri-o://a68fa3625203c296044b59b085178ebaa46e9f3b35bab2fd7b891a166a3236b9" gracePeriod=30
Nov 03 08:14:16.835638 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:16.834995    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"]
Nov 03 08:14:16.836735 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:16.836126    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd" podUID="4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" containerName="istio-proxy" containerID="cri-o://4cd7620a2ce00c0da699933e652e7de7f469e1daee40daac8fd4996a105fa128" gracePeriod=30
Nov 03 08:14:20.292442 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:20.292400    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:14:22.656830 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:14:22.645719    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:14:22.689280 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:22.673717    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t"]
Nov 03 08:14:22.802782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:22.791354    5070 generic.go:356] "Generic (PLEG): container finished" podID="2a7d0e21-8828-4ebd-a135-068d713fe575" containerID="a68fa3625203c296044b59b085178ebaa46e9f3b35bab2fd7b891a166a3236b9" exitCode=0
Nov 03 08:14:22.802782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:22.791710    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5" event={"ID":"2a7d0e21-8828-4ebd-a135-068d713fe575","Type":"ContainerDied","Data":"a68fa3625203c296044b59b085178ebaa46e9f3b35bab2fd7b891a166a3236b9"}
Nov 03 08:14:22.956275 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:22.956093    5070 generic.go:356] "Generic (PLEG): container finished" podID="4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" containerID="4cd7620a2ce00c0da699933e652e7de7f469e1daee40daac8fd4996a105fa128" exitCode=0
Nov 03 08:14:22.963933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:22.963896    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd" event={"ID":"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b","Type":"ContainerDied","Data":"4cd7620a2ce00c0da699933e652e7de7f469e1daee40daac8fd4996a105fa128"}
Nov 03 08:14:22.964189 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:22.964169    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e255cbee-dmk8t"]
Nov 03 08:14:23.165386 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.165349    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:14:23.291154 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.290480    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7d0e21-8828-4ebd-a135-068d713fe575-kube-api-access-gbqlx\" (UniqueName: \"kubernetes.io/projected/2a7d0e21-8828-4ebd-a135-068d713fe575-kube-api-access-gbqlx\") pod \"2a7d0e21-8828-4ebd-a135-068d713fe575\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") "
Nov 03 08:14:23.291154 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.290517    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-token\" (UniqueName: \"kubernetes.io/projected/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-token\") pod \"2a7d0e21-8828-4ebd-a135-068d713fe575\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") "
Nov 03 08:14:23.291154 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.290619    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-workload-socket\") pod \"2a7d0e21-8828-4ebd-a135-068d713fe575\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") "
Nov 03 08:14:23.291154 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.290662    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-workload-certs\") pod \"2a7d0e21-8828-4ebd-a135-068d713fe575\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") "
Nov 03 08:14:23.291154 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.290700    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a7d0e21-8828-4ebd-a135-068d713fe575-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7d0e21-8828-4ebd-a135-068d713fe575-istiod-ca-cert\") pod \"2a7d0e21-8828-4ebd-a135-068d713fe575\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") "
Nov 03 08:14:23.291154 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.290740    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-envoy\") pod \"2a7d0e21-8828-4ebd-a135-068d713fe575\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") "
Nov 03 08:14:23.291154 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.290906    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-podinfo\") pod \"2a7d0e21-8828-4ebd-a135-068d713fe575\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") "
Nov 03 08:14:23.291154 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.291084    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-data\") pod \"2a7d0e21-8828-4ebd-a135-068d713fe575\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") "
Nov 03 08:14:23.291154 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.291129    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-workload-socket" pod "2a7d0e21-8828-4ebd-a135-068d713fe575" (UID: "2a7d0e21-8828-4ebd-a135-068d713fe575"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:23.291154 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.291153    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-credential-socket\") pod \"2a7d0e21-8828-4ebd-a135-068d713fe575\" (UID: \"2a7d0e21-8828-4ebd-a135-068d713fe575\") "
Nov 03 08:14:23.292182 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.292024    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-workload-certs" pod "2a7d0e21-8828-4ebd-a135-068d713fe575" (UID: "2a7d0e21-8828-4ebd-a135-068d713fe575"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:23.292182 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.292057    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.292415 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.292251    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-credential-socket" pod "2a7d0e21-8828-4ebd-a135-068d713fe575" (UID: "2a7d0e21-8828-4ebd-a135-068d713fe575"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:23.292488 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.292423    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-data" pod "2a7d0e21-8828-4ebd-a135-068d713fe575" (UID: "2a7d0e21-8828-4ebd-a135-068d713fe575"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:23.292794 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.292744    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7d0e21-8828-4ebd-a135-068d713fe575-istiod-ca-cert" pod "2a7d0e21-8828-4ebd-a135-068d713fe575" (UID: "2a7d0e21-8828-4ebd-a135-068d713fe575"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:14:23.294180 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.294157    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7d0e21-8828-4ebd-a135-068d713fe575-kube-api-access-gbqlx" pod "2a7d0e21-8828-4ebd-a135-068d713fe575" (UID: "2a7d0e21-8828-4ebd-a135-068d713fe575"). InnerVolumeSpecName "kube-api-access-gbqlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:23.295177 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.295156    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-podinfo" pod "2a7d0e21-8828-4ebd-a135-068d713fe575" (UID: "2a7d0e21-8828-4ebd-a135-068d713fe575"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:14:23.295267 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.295170    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-token" pod "2a7d0e21-8828-4ebd-a135-068d713fe575" (UID: "2a7d0e21-8828-4ebd-a135-068d713fe575"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:23.296446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.296428    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:14:23.297196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.297143    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-envoy" pod "2a7d0e21-8828-4ebd-a135-068d713fe575" (UID: "2a7d0e21-8828-4ebd-a135-068d713fe575"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:23.393573 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.393535    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-token\" (UniqueName: \"kubernetes.io/projected/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-token\") pod \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") "
Nov 03 08:14:23.394271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.393602    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-credential-socket\") pod \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") "
Nov 03 08:14:23.394271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.393726    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-envoy\") pod \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") "
Nov 03 08:14:23.394271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.393841    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-workload-certs\") pod \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") "
Nov 03 08:14:23.394271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.393881    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-data\") pod \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") "
Nov 03 08:14:23.394271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.393934    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-kube-api-access-lvjz7\" (UniqueName: \"kubernetes.io/projected/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-kube-api-access-lvjz7\") pod \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") "
Nov 03 08:14:23.394271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.393976    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istiod-ca-cert\") pod \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") "
Nov 03 08:14:23.394271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.394017    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-workload-socket\") pod \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") "
Nov 03 08:14:23.394271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.394061    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-credential-socket" pod "4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" (UID: "4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:23.394271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.394213    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-data" pod "4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" (UID: "4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:23.394271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.394222    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-podinfo\") pod \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\" (UID: \"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b\") "
Nov 03 08:14:23.398950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.394661    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-workload-socket" pod "4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" (UID: "4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:23.398950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.394848    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-workload-certs" pod "4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" (UID: "4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:23.398950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.395086    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.398950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.395108    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.398950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.395122    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a7d0e21-8828-4ebd-a135-068d713fe575-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.398950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.395134    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.398950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.395147    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.398950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.395162    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.398950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.395175    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.398950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.395191    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.398950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.395205    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.398950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.395218    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a7d0e21-8828-4ebd-a135-068d713fe575-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.398950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.395232    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbqlx\" (UniqueName: \"kubernetes.io/projected/2a7d0e21-8828-4ebd-a135-068d713fe575-kube-api-access-gbqlx\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.398950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.395243    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a7d0e21-8828-4ebd-a135-068d713fe575-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.398950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.395563    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istiod-ca-cert" pod "4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" (UID: "4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:14:23.398950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.397747    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-envoy" pod "4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" (UID: "4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:23.398950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.397931    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-podinfo" pod "4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" (UID: "4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:14:23.398950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.398141    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-token" pod "4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" (UID: "4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:23.398950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.398171    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-kube-api-access-lvjz7" pod "4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" (UID: "4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b"). InnerVolumeSpecName "kube-api-access-lvjz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:23.421008 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.420981    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:14:23.499680 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.499326    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.499680 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.499387    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.499680 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.499405    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvjz7\" (UniqueName: \"kubernetes.io/projected/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-kube-api-access-lvjz7\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.499680 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.499421    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.499680 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.499434    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.600024 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.599988    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a77803-4112-45fe-bf9b-5ca069a5441f-kube-api-access-crp7h\" (UniqueName: \"kubernetes.io/projected/58a77803-4112-45fe-bf9b-5ca069a5441f-kube-api-access-crp7h\") pod \"58a77803-4112-45fe-bf9b-5ca069a5441f\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") "
Nov 03 08:14:23.600216 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.600055    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-token\" (UniqueName: \"kubernetes.io/projected/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-token\") pod \"58a77803-4112-45fe-bf9b-5ca069a5441f\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") "
Nov 03 08:14:23.600216 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.600094    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-credential-socket\") pod \"58a77803-4112-45fe-bf9b-5ca069a5441f\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") "
Nov 03 08:14:23.600216 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.600123    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-data\") pod \"58a77803-4112-45fe-bf9b-5ca069a5441f\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") "
Nov 03 08:14:23.600382 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.600262    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-envoy\") pod \"58a77803-4112-45fe-bf9b-5ca069a5441f\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") "
Nov 03 08:14:23.600382 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.600322    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a77803-4112-45fe-bf9b-5ca069a5441f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a77803-4112-45fe-bf9b-5ca069a5441f-istiod-ca-cert\") pod \"58a77803-4112-45fe-bf9b-5ca069a5441f\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") "
Nov 03 08:14:23.600382 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.600356    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-workload-socket\") pod \"58a77803-4112-45fe-bf9b-5ca069a5441f\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") "
Nov 03 08:14:23.600519 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.600437    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-credential-socket" pod "58a77803-4112-45fe-bf9b-5ca069a5441f" (UID: "58a77803-4112-45fe-bf9b-5ca069a5441f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:23.600519 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.600446    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-data" pod "58a77803-4112-45fe-bf9b-5ca069a5441f" (UID: "58a77803-4112-45fe-bf9b-5ca069a5441f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:23.600619 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.600560    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-podinfo\") pod \"58a77803-4112-45fe-bf9b-5ca069a5441f\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") "
Nov 03 08:14:23.600675 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.600633    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-workload-certs\") pod \"58a77803-4112-45fe-bf9b-5ca069a5441f\" (UID: \"58a77803-4112-45fe-bf9b-5ca069a5441f\") "
Nov 03 08:14:23.600770 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.600733    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-workload-socket" pod "58a77803-4112-45fe-bf9b-5ca069a5441f" (UID: "58a77803-4112-45fe-bf9b-5ca069a5441f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:23.600838 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.600780    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a77803-4112-45fe-bf9b-5ca069a5441f-istiod-ca-cert" pod "58a77803-4112-45fe-bf9b-5ca069a5441f" (UID: "58a77803-4112-45fe-bf9b-5ca069a5441f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:14:23.601025 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.600949    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-workload-certs" pod "58a77803-4112-45fe-bf9b-5ca069a5441f" (UID: "58a77803-4112-45fe-bf9b-5ca069a5441f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:23.601666 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.601640    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58a77803-4112-45fe-bf9b-5ca069a5441f-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.601814 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.601672    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.601814 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.601688    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.601814 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.601702    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.601814 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.601715    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.602745 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.602688    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a77803-4112-45fe-bf9b-5ca069a5441f-kube-api-access-crp7h" pod "58a77803-4112-45fe-bf9b-5ca069a5441f" (UID: "58a77803-4112-45fe-bf9b-5ca069a5441f"). InnerVolumeSpecName "kube-api-access-crp7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:23.602745 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.602704    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-podinfo" pod "58a77803-4112-45fe-bf9b-5ca069a5441f" (UID: "58a77803-4112-45fe-bf9b-5ca069a5441f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:14:23.602970 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.602843    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-token" pod "58a77803-4112-45fe-bf9b-5ca069a5441f" (UID: "58a77803-4112-45fe-bf9b-5ca069a5441f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:23.604587 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.604562    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-envoy" pod "58a77803-4112-45fe-bf9b-5ca069a5441f" (UID: "58a77803-4112-45fe-bf9b-5ca069a5441f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:23.701281 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.701241    5070 scope.go:122] "RemoveContainer" containerID="00ac02669fdb0661440f0185da90b3a51d6cd62fe74be834b83dbfb161fc09ba"
Nov 03 08:14:23.702953 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.702928    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crp7h\" (UniqueName: \"kubernetes.io/projected/58a77803-4112-45fe-bf9b-5ca069a5441f-kube-api-access-crp7h\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.702953 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.702954    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.703148 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.702964    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.703148 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.702974    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58a77803-4112-45fe-bf9b-5ca069a5441f-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:23.715577 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.715552    5070 scope.go:122] "RemoveContainer" containerID="15c0b1253b5930b6f2cd7a09b6088dd5d384ff98edb7572fc526400c717b7d20"
Nov 03 08:14:23.729136 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.729119    5070 scope.go:122] "RemoveContainer" containerID="551d089071bdf3e3e3a2da21b68c36436a882bb045fdc5a5974cfdca88dc1174"
Nov 03 08:14:23.741101 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.741071    5070 scope.go:122] "RemoveContainer" containerID="f777a77aa63ca4bb1fe3a81b77eeef2d45346bc68e85a2f01a5255c0bccaf3eb"
Nov 03 08:14:23.753995 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.753968    5070 scope.go:122] "RemoveContainer" containerID="a68fa3625203c296044b59b085178ebaa46e9f3b35bab2fd7b891a166a3236b9"
Nov 03 08:14:23.765637 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.765617    5070 scope.go:122] "RemoveContainer" containerID="82a2dca6a9eca705a350bb91693d8488079f4502485f617b33466a0b6902ae42"
Nov 03 08:14:23.779292 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.779269    5070 scope.go:122] "RemoveContainer" containerID="db54f21a277799a23b445ad5823b6b73e5525664bbf9b404e0ca28d0b8c5ad89"
Nov 03 08:14:23.791213 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.791196    5070 scope.go:122] "RemoveContainer" containerID="97f795f398132977c14fd2f64b5399a3a3a2f2d7730c616628529acc52ae721f"
Nov 03 08:14:23.802687 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.802668    5070 scope.go:122] "RemoveContainer" containerID="4cd7620a2ce00c0da699933e652e7de7f469e1daee40daac8fd4996a105fa128"
Nov 03 08:14:23.814235 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.814214    5070 scope.go:122] "RemoveContainer" containerID="16c0392664a0395133f1a3c7b4da5f0db6d46141930d1bd32f0b1b3e562e712d"
Nov 03 08:14:23.825462 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.825441    5070 scope.go:122] "RemoveContainer" containerID="9cb8e860a4d04f080559fdf7246c2c01849e61370c90cfc4d54e019669734ce0"
Nov 03 08:14:23.835796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.835739    5070 scope.go:122] "RemoveContainer" containerID="8dda71f828b1aa827dbf5f149fefffcb7f07b97d7a0a8d122039295a2d8ecdce"
Nov 03 08:14:23.849574 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.848736    5070 scope.go:122] "RemoveContainer" containerID="c1b6d966650e3e097ce2a8b435c86ecd14a9cd040431c5b8828678f5f834b870"
Nov 03 08:14:23.866395 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.865717    5070 scope.go:122] "RemoveContainer" containerID="a240c9848b03ebb1243e847f66536317265dfff5bb635e971836523d2dbebd2b"
Nov 03 08:14:23.876678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.876651    5070 scope.go:122] "RemoveContainer" containerID="00f2351acc831976a876ea110f1c822dc134cb870bd18625b375bdabae409fcb"
Nov 03 08:14:23.888088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:23.888067    5070 scope.go:122] "RemoveContainer" containerID="240e1d97c6b26eb6ae368e594992aaed38e832fed1d5f47a3b604056638d7c36"
Nov 03 08:14:24.041196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:24.040416    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"
Nov 03 08:14:24.120066 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:24.120035    5070 generic.go:356] "Generic (PLEG): container finished" podID="58a77803-4112-45fe-bf9b-5ca069a5441f" containerID="71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997" exitCode=0
Nov 03 08:14:24.122212 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:24.120277    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"
Nov 03 08:14:24.146506 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:24.142840    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"
Nov 03 08:14:24.158200 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:24.158180    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3866ec89-7dfb-43db-acf0-ea0c3069c989" path="/var/lib/kubelet/pods/3866ec89-7dfb-43db-acf0-ea0c3069c989/volumes"
Nov 03 08:14:24.165975 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:24.165950    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd" event={"ID":"4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b","Type":"ContainerDied","Data":"ccebbb56272b2ed2808cd715a2b58f4d90fe106e7ad52df0a9af04ef35db2bcf"}
Nov 03 08:14:24.174206 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:24.174183    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr" event={"ID":"58a77803-4112-45fe-bf9b-5ca069a5441f","Type":"ContainerDied","Data":"71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997"}
Nov 03 08:14:24.174508 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:24.174482    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr" event={"ID":"58a77803-4112-45fe-bf9b-5ca069a5441f","Type":"ContainerDied","Data":"c09bf042184b873b27f9384a6cb6afa64e7b9feda426001cea74ab3b16c0080d"}
Nov 03 08:14:24.174718 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:24.174703    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5" event={"ID":"2a7d0e21-8828-4ebd-a135-068d713fe575","Type":"ContainerDied","Data":"83cc0229870dcfe5d1463b5c520d2899734ec8b7c0a4f2b3ac84fb94225b67fd"}
Nov 03 08:14:24.174973 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:24.174958    5070 scope.go:122] "RemoveContainer" containerID="71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997"
Nov 03 08:14:24.188902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:24.188880    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"]
Nov 03 08:14:24.209192 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:24.208653    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-tbcbd"]
Nov 03 08:14:24.221304 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:24.221279    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"]
Nov 03 08:14:24.229812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:24.229787    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-tdqk5"]
Nov 03 08:14:24.253839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:24.253812    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"]
Nov 03 08:14:24.257429 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:24.257408    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-dfb883-openshift-default-7745bff66f-gvqdr"]
Nov 03 08:14:24.470236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:24.469889    5070 scope.go:122] "RemoveContainer" containerID="0df6c92f2ed4ae98ac879da0364e52048aa4c49e89ea406ac019138d1ae36d7a"
Nov 03 08:14:24.646785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:24.646746    5070 scope.go:122] "RemoveContainer" containerID="71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997"
Nov 03 08:14:24.647908 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:14:24.647637    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997\": container with ID starting with 71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997 not found: ID does not exist" containerID="71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997"
Nov 03 08:14:24.647908 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:24.647675    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997"} err="failed to get container status \"71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997\": rpc error: code = NotFound desc = could not find container \"71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997\": container with ID starting with 71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997 not found: ID does not exist"
Nov 03 08:14:24.647908 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:24.647701    5070 scope.go:122] "RemoveContainer" containerID="0df6c92f2ed4ae98ac879da0364e52048aa4c49e89ea406ac019138d1ae36d7a"
Nov 03 08:14:24.658888 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:14:24.658858    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0df6c92f2ed4ae98ac879da0364e52048aa4c49e89ea406ac019138d1ae36d7a\": container with ID starting with 0df6c92f2ed4ae98ac879da0364e52048aa4c49e89ea406ac019138d1ae36d7a not found: ID does not exist" containerID="0df6c92f2ed4ae98ac879da0364e52048aa4c49e89ea406ac019138d1ae36d7a"
Nov 03 08:14:24.659044 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:24.659023    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df6c92f2ed4ae98ac879da0364e52048aa4c49e89ea406ac019138d1ae36d7a"} err="failed to get container status \"0df6c92f2ed4ae98ac879da0364e52048aa4c49e89ea406ac019138d1ae36d7a\": rpc error: code = NotFound desc = could not find container \"0df6c92f2ed4ae98ac879da0364e52048aa4c49e89ea406ac019138d1ae36d7a\": container with ID starting with 0df6c92f2ed4ae98ac879da0364e52048aa4c49e89ea406ac019138d1ae36d7a not found: ID does not exist"
Nov 03 08:14:26.028952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:26.028645    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a7d0e21-8828-4ebd-a135-068d713fe575" path="/var/lib/kubelet/pods/2a7d0e21-8828-4ebd-a135-068d713fe575/volumes"
Nov 03 08:14:26.029470 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:26.029445    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" path="/var/lib/kubelet/pods/4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b/volumes"
Nov 03 08:14:26.030210 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:26.030185    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58a77803-4112-45fe-bf9b-5ca069a5441f" path="/var/lib/kubelet/pods/58a77803-4112-45fe-bf9b-5ca069a5441f/volumes"
Nov 03 08:14:27.061069 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:27.061029    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d"]
Nov 03 08:14:27.065625 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:27.065595    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb887217-g6x8d"]
Nov 03 08:14:28.027901 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:28.027871    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43306943-c1e8-4bb2-be03-c540de81d41d" path="/var/lib/kubelet/pods/43306943-c1e8-4bb2-be03-c540de81d41d/volumes"
Nov 03 08:14:30.293875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:30.293843    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"]
Nov 03 08:14:30.294471 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:30.294089    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx" podUID="fd8f468d-ba76-4155-85d8-7952b0cd31f5" containerName="istio-proxy" containerID="cri-o://4bfb76d6582f248ba0b56bdb29e83acc63f19532dd9f8cad7ab54f0462f6e212" gracePeriod=30
Nov 03 08:14:30.779509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:30.779464    5070 patch_prober.go:29] interesting pod/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:14:30.779667 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:30.779526    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx" podUID="fd8f468d-ba76-4155-85d8-7952b0cd31f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:14:32.748754 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:14:32.748712    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77803_4112_45fe_bf9b_5ca069a5441f.slice/crio-71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77803_4112_45fe_bf9b_5ca069a5441f.slice/crio-conmon-71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 08:14:35.595050 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.594964    5070 generic.go:356] "Generic (PLEG): container finished" podID="fd8f468d-ba76-4155-85d8-7952b0cd31f5" containerID="4bfb76d6582f248ba0b56bdb29e83acc63f19532dd9f8cad7ab54f0462f6e212" exitCode=0
Nov 03 08:14:35.595489 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.595043    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx" event={"ID":"fd8f468d-ba76-4155-85d8-7952b0cd31f5","Type":"ContainerDied","Data":"4bfb76d6582f248ba0b56bdb29e83acc63f19532dd9f8cad7ab54f0462f6e212"}
Nov 03 08:14:35.595489 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.595132    5070 scope.go:122] "RemoveContainer" containerID="d584251cb2ab18dd63b570b1b17083630b0e090a483250046f95978238188b7b"
Nov 03 08:14:35.683926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.683893    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:14:35.793905 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.793869    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-podinfo\") pod \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") "
Nov 03 08:14:35.794090 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.793937    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-credential-socket\") pod \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") "
Nov 03 08:14:35.794090 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.793973    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istiod-ca-cert\") pod \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") "
Nov 03 08:14:35.794090 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.794059    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-data\") pod \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") "
Nov 03 08:14:35.794225 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.794178    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-token\" (UniqueName: \"kubernetes.io/projected/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-token\") pod \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") "
Nov 03 08:14:35.794260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.794253    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8f468d-ba76-4155-85d8-7952b0cd31f5-kube-api-access-2mq5n\" (UniqueName: \"kubernetes.io/projected/fd8f468d-ba76-4155-85d8-7952b0cd31f5-kube-api-access-2mq5n\") pod \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") "
Nov 03 08:14:35.794373 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.794290    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-envoy\") pod \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") "
Nov 03 08:14:35.795192 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.794405    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-workload-socket\") pod \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") "
Nov 03 08:14:35.795192 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.794507    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-workload-certs\") pod \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\" (UID: \"fd8f468d-ba76-4155-85d8-7952b0cd31f5\") "
Nov 03 08:14:35.796022 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.795987    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-workload-certs" pod "fd8f468d-ba76-4155-85d8-7952b0cd31f5" (UID: "fd8f468d-ba76-4155-85d8-7952b0cd31f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:35.799946 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.799918    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istiod-ca-cert" pod "fd8f468d-ba76-4155-85d8-7952b0cd31f5" (UID: "fd8f468d-ba76-4155-85d8-7952b0cd31f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:14:35.800363 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.800134    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-workload-socket" pod "fd8f468d-ba76-4155-85d8-7952b0cd31f5" (UID: "fd8f468d-ba76-4155-85d8-7952b0cd31f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:35.800456 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.800377    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-credential-socket" pod "fd8f468d-ba76-4155-85d8-7952b0cd31f5" (UID: "fd8f468d-ba76-4155-85d8-7952b0cd31f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:35.800606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.800542    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-data" pod "fd8f468d-ba76-4155-85d8-7952b0cd31f5" (UID: "fd8f468d-ba76-4155-85d8-7952b0cd31f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:35.802051 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.802025    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-envoy" pod "fd8f468d-ba76-4155-85d8-7952b0cd31f5" (UID: "fd8f468d-ba76-4155-85d8-7952b0cd31f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:14:35.803581 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.803550    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-podinfo" pod "fd8f468d-ba76-4155-85d8-7952b0cd31f5" (UID: "fd8f468d-ba76-4155-85d8-7952b0cd31f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:14:35.803687 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.803596    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-token" pod "fd8f468d-ba76-4155-85d8-7952b0cd31f5" (UID: "fd8f468d-ba76-4155-85d8-7952b0cd31f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:35.804214 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.804192    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8f468d-ba76-4155-85d8-7952b0cd31f5-kube-api-access-2mq5n" pod "fd8f468d-ba76-4155-85d8-7952b0cd31f5" (UID: "fd8f468d-ba76-4155-85d8-7952b0cd31f5"). InnerVolumeSpecName "kube-api-access-2mq5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:35.897561 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.897531    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:35.897561 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.897559    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:35.897829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.897568    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:35.897829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.897578    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mq5n\" (UniqueName: \"kubernetes.io/projected/fd8f468d-ba76-4155-85d8-7952b0cd31f5-kube-api-access-2mq5n\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:35.897829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.897588    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:35.897829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.897597    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:35.897829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.897605    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:35.897829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.897614    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8f468d-ba76-4155-85d8-7952b0cd31f5-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:35.897829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:35.897623    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8f468d-ba76-4155-85d8-7952b0cd31f5-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:36.604353 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:36.604324    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"
Nov 03 08:14:36.604881 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:36.604365    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx" event={"ID":"fd8f468d-ba76-4155-85d8-7952b0cd31f5","Type":"ContainerDied","Data":"9057310183bfe9435fd56557cbdb8cf2aea31bf84527f58410da910067a8a37f"}
Nov 03 08:14:36.604881 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:36.604430    5070 scope.go:122] "RemoveContainer" containerID="4bfb76d6582f248ba0b56bdb29e83acc63f19532dd9f8cad7ab54f0462f6e212"
Nov 03 08:14:36.627656 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:36.627629    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"]
Nov 03 08:14:36.633542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:36.633518    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e255cbee-openshift-default-774b46d7c-q4bdx"]
Nov 03 08:14:37.799925 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:14:37.799891    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77803_4112_45fe_bf9b_5ca069a5441f.slice/crio-71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77803_4112_45fe_bf9b_5ca069a5441f.slice/crio-conmon-71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 08:14:38.027737 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:38.027696    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd8f468d-ba76-4155-85d8-7952b0cd31f5" path="/var/lib/kubelet/pods/fd8f468d-ba76-4155-85d8-7952b0cd31f5/volumes"
Nov 03 08:14:42.849316 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:14:42.849266    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77803_4112_45fe_bf9b_5ca069a5441f.slice/crio-conmon-71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77803_4112_45fe_bf9b_5ca069a5441f.slice/crio-71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 08:14:44.138394 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.138364    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"]
Nov 03 08:14:44.140418 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.140393    5070 cpu_manager.go:414] "containerMap: removing container" podUID="43306943-c1e8-4bb2-be03-c540de81d41d" containerName="revoke-keys"
Nov 03 08:14:44.140418 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.140416    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43306943-c1e8-4bb2-be03-c540de81d41d" containerName="revoke-keys"
Nov 03 08:14:44.140622 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.140449    5070 cpu_manager.go:414] "containerMap: removing container" podUID="58a77803-4112-45fe-bf9b-5ca069a5441f" containerName="istio-proxy"
Nov 03 08:14:44.140622 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.140463    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a77803-4112-45fe-bf9b-5ca069a5441f" containerName="istio-proxy"
Nov 03 08:14:44.140622 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.140491    5070 cpu_manager.go:414] "containerMap: removing container" podUID="fd8f468d-ba76-4155-85d8-7952b0cd31f5" containerName="istio-proxy"
Nov 03 08:14:44.140622 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.140499    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8f468d-ba76-4155-85d8-7952b0cd31f5" containerName="istio-proxy"
Nov 03 08:14:44.140622 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.140525    5070 cpu_manager.go:414] "containerMap: removing container" podUID="2a7d0e21-8828-4ebd-a135-068d713fe575" containerName="istio-proxy"
Nov 03 08:14:44.140622 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.140532    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a7d0e21-8828-4ebd-a135-068d713fe575" containerName="istio-proxy"
Nov 03 08:14:44.140622 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.140546    5070 cpu_manager.go:414] "containerMap: removing container" podUID="fd8f468d-ba76-4155-85d8-7952b0cd31f5" containerName="istio-proxy"
Nov 03 08:14:44.140622 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.140552    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8f468d-ba76-4155-85d8-7952b0cd31f5" containerName="istio-proxy"
Nov 03 08:14:44.140622 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.140619    5070 cpu_manager.go:414] "containerMap: removing container" podUID="3866ec89-7dfb-43db-acf0-ea0c3069c989" containerName="revoke-keys"
Nov 03 08:14:44.141088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.140628    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3866ec89-7dfb-43db-acf0-ea0c3069c989" containerName="revoke-keys"
Nov 03 08:14:44.141088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.140647    5070 cpu_manager.go:414] "containerMap: removing container" podUID="4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" containerName="istio-proxy"
Nov 03 08:14:44.141088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.140655    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" containerName="istio-proxy"
Nov 03 08:14:44.141088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.140890    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="43306943-c1e8-4bb2-be03-c540de81d41d" containerName="revoke-keys"
Nov 03 08:14:44.141088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.140910    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a77803-4112-45fe-bf9b-5ca069a5441f" containerName="istio-proxy"
Nov 03 08:14:44.141088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.140923    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8f468d-ba76-4155-85d8-7952b0cd31f5" containerName="istio-proxy"
Nov 03 08:14:44.141088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.140936    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a7d0e21-8828-4ebd-a135-068d713fe575" containerName="istio-proxy"
Nov 03 08:14:44.141088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.140947    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a77803-4112-45fe-bf9b-5ca069a5441f" containerName="istio-proxy"
Nov 03 08:14:44.141088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.140961    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="3866ec89-7dfb-43db-acf0-ea0c3069c989" containerName="revoke-keys"
Nov 03 08:14:44.141088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.140973    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="4807bcf5-81e2-4946-8c7e-06b1a7b2fd2b" containerName="istio-proxy"
Nov 03 08:14:44.147857 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.147470    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.162778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.162691    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"]
Nov 03 08:14:44.163151 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.163108    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9b559e-openshift-default-dockercfg-6hqmb\""
Nov 03 08:14:44.345782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.339077    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acaf418a-affb-4714-a5ce-5673ed5be68e-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.345782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.339161    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.345782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.339206    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.345782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.339248    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.345782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.339328    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.345782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.339385    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.345782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.339470    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.345782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.339506    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnkh2\" (UniqueName: \"kubernetes.io/projected/acaf418a-affb-4714-a5ce-5673ed5be68e-kube-api-access-lnkh2\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.345782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.339643    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.441823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.441108    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.441823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.441173    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnkh2\" (UniqueName: \"kubernetes.io/projected/acaf418a-affb-4714-a5ce-5673ed5be68e-kube-api-access-lnkh2\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.441823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.441297    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.441823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.441468    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acaf418a-affb-4714-a5ce-5673ed5be68e-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.441823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.441522    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.441823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.441556    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.441823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.441594    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.441823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.441661    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.441823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.441704    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.443357 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.442692    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.443357 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.443316    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.454777 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.451940    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acaf418a-affb-4714-a5ce-5673ed5be68e-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.454777 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.454221    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.455157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.455128    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.467643 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.467613    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.500785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.487640    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.500785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.500353    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnkh2\" (UniqueName: \"kubernetes.io/projected/acaf418a-affb-4714-a5ce-5673ed5be68e-kube-api-access-lnkh2\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.501026 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.500991    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:44.774949 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:44.774862    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:45.086333 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:14:45.086308    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacaf418a_affb_4714_a5ce_5673ed5be68e.slice/crio-6a9ce820723071620e4fdef5154b561b53e3e39b3679dcc7badbca8eb5481fd8 WatchSource:0}: Error finding container 6a9ce820723071620e4fdef5154b561b53e3e39b3679dcc7badbca8eb5481fd8: Status 404 returned error can't find the container with id 6a9ce820723071620e4fdef5154b561b53e3e39b3679dcc7badbca8eb5481fd8
Nov 03 08:14:45.089060 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:45.089026    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:14:45.089157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:45.089094    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:14:45.089157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:45.089123    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:14:45.090313 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:45.090286    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"]
Nov 03 08:14:45.767910 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:45.767873    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" event={"ID":"acaf418a-affb-4714-a5ce-5673ed5be68e","Type":"ContainerStarted","Data":"9bd4a10841b8474e8baebcb447545c93c6cb1b79d710ac3e880d8227edf1f286"}
Nov 03 08:14:45.769251 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:45.767925    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" event={"ID":"acaf418a-affb-4714-a5ce-5673ed5be68e","Type":"ContainerStarted","Data":"6a9ce820723071620e4fdef5154b561b53e3e39b3679dcc7badbca8eb5481fd8"}
Nov 03 08:14:45.775592 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:45.775570    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:45.785602 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:45.785561    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" podStartSLOduration=1.7855479239999998 podStartE2EDuration="1.785547924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:45.783964725 +0000 UTC m=+1283.898428656" watchObservedRunningTime="2026-11-03 08:14:45.785547924 +0000 UTC m=+1283.900011855"
Nov 03 08:14:46.781096 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:46.781070    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:47.784276 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:47.784241    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:47.785026 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:47.785007    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:14:48.977449 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:48.977411    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt"]
Nov 03 08:14:48.978353 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:48.978328    5070 cpu_manager.go:414] "containerMap: removing container" podUID="58a77803-4112-45fe-bf9b-5ca069a5441f" containerName="istio-proxy"
Nov 03 08:14:48.978353 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:48.978353    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a77803-4112-45fe-bf9b-5ca069a5441f" containerName="istio-proxy"
Nov 03 08:14:48.978585 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:48.978570    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8f468d-ba76-4155-85d8-7952b0cd31f5" containerName="istio-proxy"
Nov 03 08:14:48.982728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:48.982695    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt"
Nov 03 08:14:48.985909 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:48.985886    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:14:48.986746 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:48.985888    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:14:48.986746 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:48.986375    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:14:48.986746 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:48.986389    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:14:48.989894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:48.989870    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt"]
Nov 03 08:14:49.100437 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:49.100411    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a00028a-af52-4a83-b3d6-176671717fd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt\" (UID: \"1a00028a-af52-4a83-b3d6-176671717fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt"
Nov 03 08:14:49.100583 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:49.100442    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a00028a-af52-4a83-b3d6-176671717fd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt\" (UID: \"1a00028a-af52-4a83-b3d6-176671717fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt"
Nov 03 08:14:49.100583 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:49.100573    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bbw8\" (UniqueName: \"kubernetes.io/projected/1a00028a-af52-4a83-b3d6-176671717fd7-kube-api-access-5bbw8\") pod \"maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt\" (UID: \"1a00028a-af52-4a83-b3d6-176671717fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt"
Nov 03 08:14:49.202195 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:49.202167    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a00028a-af52-4a83-b3d6-176671717fd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt\" (UID: \"1a00028a-af52-4a83-b3d6-176671717fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt"
Nov 03 08:14:49.202335 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:49.202208    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a00028a-af52-4a83-b3d6-176671717fd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt\" (UID: \"1a00028a-af52-4a83-b3d6-176671717fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt"
Nov 03 08:14:49.202409 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:49.202339    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bbw8\" (UniqueName: \"kubernetes.io/projected/1a00028a-af52-4a83-b3d6-176671717fd7-kube-api-access-5bbw8\") pod \"maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt\" (UID: \"1a00028a-af52-4a83-b3d6-176671717fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt"
Nov 03 08:14:49.202886 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:49.202861    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a00028a-af52-4a83-b3d6-176671717fd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt\" (UID: \"1a00028a-af52-4a83-b3d6-176671717fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt"
Nov 03 08:14:49.204836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:49.204815    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a00028a-af52-4a83-b3d6-176671717fd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt\" (UID: \"1a00028a-af52-4a83-b3d6-176671717fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt"
Nov 03 08:14:49.209599 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:49.209577    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bbw8\" (UniqueName: \"kubernetes.io/projected/1a00028a-af52-4a83-b3d6-176671717fd7-kube-api-access-5bbw8\") pod \"maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt\" (UID: \"1a00028a-af52-4a83-b3d6-176671717fd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt"
Nov 03 08:14:49.300524 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:49.300468    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt"
Nov 03 08:14:49.594812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:49.594666    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt"]
Nov 03 08:14:49.982874 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:49.982819    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt" event={"ID":"1a00028a-af52-4a83-b3d6-176671717fd7","Type":"ContainerStarted","Data":"6df0be1d269595c7aadf2ae8fd33c3283ec842aaf7fb4bd11f6f8aa772ec756a"}
Nov 03 08:14:50.990245 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:50.990201    5070 generic.go:356] "Generic (PLEG): container finished" podID="1a00028a-af52-4a83-b3d6-176671717fd7" containerID="814bea5c0fbef0170dce997f3c620cc6c2902cf5aca0dd02f29ee7e4d25ab9c4" exitCode=0
Nov 03 08:14:50.990716 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:50.990316    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt" event={"ID":"1a00028a-af52-4a83-b3d6-176671717fd7","Type":"ContainerDied","Data":"814bea5c0fbef0170dce997f3c620cc6c2902cf5aca0dd02f29ee7e4d25ab9c4"}
Nov 03 08:14:51.220222 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:14:51.220134    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77803_4112_45fe_bf9b_5ca069a5441f.slice/crio-conmon-71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77803_4112_45fe_bf9b_5ca069a5441f.slice/crio-71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 08:14:51.220383 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:14:51.220234    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77803_4112_45fe_bf9b_5ca069a5441f.slice/crio-71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77803_4112_45fe_bf9b_5ca069a5441f.slice/crio-conmon-71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 08:14:52.059207 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:14:52.059179    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:14:52.202158 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:52.202136    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt"
Nov 03 08:14:52.267571 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:52.267498    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a00028a-af52-4a83-b3d6-176671717fd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a00028a-af52-4a83-b3d6-176671717fd7-maas-api-xks-ca\") pod \"1a00028a-af52-4a83-b3d6-176671717fd7\" (UID: \"1a00028a-af52-4a83-b3d6-176671717fd7\") "
Nov 03 08:14:52.267722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:52.267695    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a00028a-af52-4a83-b3d6-176671717fd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a00028a-af52-4a83-b3d6-176671717fd7-maas-api-openshift-ca\") pod \"1a00028a-af52-4a83-b3d6-176671717fd7\" (UID: \"1a00028a-af52-4a83-b3d6-176671717fd7\") "
Nov 03 08:14:52.267808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:52.267738    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a00028a-af52-4a83-b3d6-176671717fd7-kube-api-access-5bbw8\" (UniqueName: \"kubernetes.io/projected/1a00028a-af52-4a83-b3d6-176671717fd7-kube-api-access-5bbw8\") pod \"1a00028a-af52-4a83-b3d6-176671717fd7\" (UID: \"1a00028a-af52-4a83-b3d6-176671717fd7\") "
Nov 03 08:14:52.268138 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:52.268108    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a00028a-af52-4a83-b3d6-176671717fd7-maas-api-openshift-ca" pod "1a00028a-af52-4a83-b3d6-176671717fd7" (UID: "1a00028a-af52-4a83-b3d6-176671717fd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:14:52.268415 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:52.268395    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a00028a-af52-4a83-b3d6-176671717fd7-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:52.269955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:52.269923    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a00028a-af52-4a83-b3d6-176671717fd7-kube-api-access-5bbw8" pod "1a00028a-af52-4a83-b3d6-176671717fd7" (UID: "1a00028a-af52-4a83-b3d6-176671717fd7"). InnerVolumeSpecName "kube-api-access-5bbw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:14:52.270145 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:52.270111    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a00028a-af52-4a83-b3d6-176671717fd7-maas-api-xks-ca" pod "1a00028a-af52-4a83-b3d6-176671717fd7" (UID: "1a00028a-af52-4a83-b3d6-176671717fd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:14:52.369725 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:52.369694    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a00028a-af52-4a83-b3d6-176671717fd7-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:52.369725 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:52.369723    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bbw8\" (UniqueName: \"kubernetes.io/projected/1a00028a-af52-4a83-b3d6-176671717fd7-kube-api-access-5bbw8\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:14:52.728926 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:14:52.728887    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77803_4112_45fe_bf9b_5ca069a5441f.slice/crio-conmon-71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77803_4112_45fe_bf9b_5ca069a5441f.slice/crio-71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 08:14:52.869630 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:14:52.869596    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77803_4112_45fe_bf9b_5ca069a5441f.slice/crio-71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77803_4112_45fe_bf9b_5ca069a5441f.slice/crio-conmon-71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 08:14:53.008601 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:53.008533    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt"
Nov 03 08:14:53.008738 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:53.008537    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt" event={"ID":"1a00028a-af52-4a83-b3d6-176671717fd7","Type":"ContainerDied","Data":"6df0be1d269595c7aadf2ae8fd33c3283ec842aaf7fb4bd11f6f8aa772ec756a"}
Nov 03 08:14:53.008738 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:53.008654    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6df0be1d269595c7aadf2ae8fd33c3283ec842aaf7fb4bd11f6f8aa772ec756a"
Nov 03 08:14:55.677842 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.677807    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"]
Nov 03 08:14:55.679341 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.679308    5070 cpu_manager.go:414] "containerMap: removing container" podUID="1a00028a-af52-4a83-b3d6-176671717fd7" containerName="revoke-keys"
Nov 03 08:14:55.679516 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.679500    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a00028a-af52-4a83-b3d6-176671717fd7" containerName="revoke-keys"
Nov 03 08:14:55.679868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.679850    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a00028a-af52-4a83-b3d6-176671717fd7" containerName="revoke-keys"
Nov 03 08:14:55.684031 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.684010    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.688493 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.688471    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e0f576-openshift-default-dockercfg-wnxjc\""
Nov 03 08:14:55.705155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.705134    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"]
Nov 03 08:14:55.750909 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.750880    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-workload-certs\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.751088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.751072    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-envoy\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.751192 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.751099    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-podinfo\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.751280 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.751254    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-token\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.751371 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.751282    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-workload-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.751371 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.751340    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-data\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.751485 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.751416    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istiod-ca-cert\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.751538 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.751520    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-credential-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.751634 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.751611    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hc4f\" (UniqueName: \"kubernetes.io/projected/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-kube-api-access-6hc4f\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.862958 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.854165    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hc4f\" (UniqueName: \"kubernetes.io/projected/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-kube-api-access-6hc4f\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.863556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.863517    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-workload-certs\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.863797 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.863776    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-envoy\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.883889 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.883850    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-podinfo\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.884301 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.884276    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-token\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.884446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.884430    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-workload-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.884592 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.884578    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-data\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.884709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.884691    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istiod-ca-cert\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.884972 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.884952    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-credential-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.885728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.885704    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-credential-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.886480 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.886458    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-workload-certs\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.887469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.887447    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-workload-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.888474 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.888452    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-data\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.915785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.908056    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istiod-ca-cert\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.950910 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.945080    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-envoy\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:55.989794 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.978032    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hc4f\" (UniqueName: \"kubernetes.io/projected/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-kube-api-access-6hc4f\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:56.007839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.991415    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-podinfo\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:56.007839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:55.995601    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-token\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:56.030282 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:56.014059    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:56.610849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:56.603933    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"]
Nov 03 08:14:56.621801 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:56.621295    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:14:56.621801 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:56.621395    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:14:56.621801 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:56.621444    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:14:57.227184 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:57.227148    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" event={"ID":"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6","Type":"ContainerStarted","Data":"5c89f506e79ac82b260040c7148b1e731c64138a5485a246e28a4ad6a3fb01d1"}
Nov 03 08:14:57.227602 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:57.227203    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" event={"ID":"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6","Type":"ContainerStarted","Data":"0c20e43ecdb4f45981b80729d7e0bda1ffc98591e3fffd7d713517dbdd4708d6"}
Nov 03 08:14:57.244416 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:57.244375    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" podStartSLOduration=2.244350331 podStartE2EDuration="2.244350331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:14:57.244227829 +0000 UTC m=+1295.358691760" watchObservedRunningTime="2026-11-03 08:14:57.244350331 +0000 UTC m=+1295.358814259"
Nov 03 08:14:58.035817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:58.035745    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:58.036019 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:58.035940    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:58.235322 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:58.235290    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:14:58.236075 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:14:58.236056    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:15:00.160793 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.160361    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg"]
Nov 03 08:15:00.168031 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.167496    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg"
Nov 03 08:15:00.188971 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.183057    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:15:00.188971 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.183291    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:15:00.188971 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.183412    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:15:00.211269 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.207913    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b"]
Nov 03 08:15:00.215105 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.214249    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b"
Nov 03 08:15:00.227688 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.224868    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm"]
Nov 03 08:15:00.235083 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.235060    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm"
Nov 03 08:15:00.283606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.283563    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt"]
Nov 03 08:15:00.298825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.294715    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt2rk\" (UniqueName: \"kubernetes.io/projected/8a05958e-20e4-4050-a9a8-76c22342d517-kube-api-access-pt2rk\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg\" (UID: \"8a05958e-20e4-4050-a9a8-76c22342d517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg"
Nov 03 08:15:00.298825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.294782    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sg8p\" (UniqueName: \"kubernetes.io/projected/e6086e8f-a1b6-43e2-8a92-3a962774083c-kube-api-access-9sg8p\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm\" (UID: \"e6086e8f-a1b6-43e2-8a92-3a962774083c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm"
Nov 03 08:15:00.298825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.294948    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgqk4\" (UniqueName: \"kubernetes.io/projected/6923d57c-6c82-4fe8-bd99-15e3be887ce1-kube-api-access-sgqk4\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b\" (UID: \"6923d57c-6c82-4fe8-bd99-15e3be887ce1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b"
Nov 03 08:15:00.298825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.295073    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a05958e-20e4-4050-a9a8-76c22342d517-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg\" (UID: \"8a05958e-20e4-4050-a9a8-76c22342d517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg"
Nov 03 08:15:00.298825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.295193    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6086e8f-a1b6-43e2-8a92-3a962774083c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm\" (UID: \"e6086e8f-a1b6-43e2-8a92-3a962774083c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm"
Nov 03 08:15:00.298825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.295320    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6923d57c-6c82-4fe8-bd99-15e3be887ce1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b\" (UID: \"6923d57c-6c82-4fe8-bd99-15e3be887ce1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b"
Nov 03 08:15:00.324940 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.321942    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg"]
Nov 03 08:15:00.338830 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.330378    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg"]
Nov 03 08:15:00.338830 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.330425    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt"]
Nov 03 08:15:00.338830 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.330451    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b"]
Nov 03 08:15:00.338830 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.330546    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg"
Nov 03 08:15:00.340837 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.339110    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt"
Nov 03 08:15:00.395824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.381590    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg"]
Nov 03 08:15:00.395824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.387890    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd"]
Nov 03 08:15:00.395824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.395488    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd"
Nov 03 08:15:00.405004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.397220    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sg8p\" (UniqueName: \"kubernetes.io/projected/e6086e8f-a1b6-43e2-8a92-3a962774083c-kube-api-access-9sg8p\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm\" (UID: \"e6086e8f-a1b6-43e2-8a92-3a962774083c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm"
Nov 03 08:15:00.405004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.397292    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmv6j\" (UniqueName: \"kubernetes.io/projected/f228239e-f1d7-4bb8-9fc9-edc630a9df41-kube-api-access-vmv6j\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt\" (UID: \"f228239e-f1d7-4bb8-9fc9-edc630a9df41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt"
Nov 03 08:15:00.405004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.397448    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f228239e-f1d7-4bb8-9fc9-edc630a9df41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt\" (UID: \"f228239e-f1d7-4bb8-9fc9-edc630a9df41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt"
Nov 03 08:15:00.405004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.397520    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgqk4\" (UniqueName: \"kubernetes.io/projected/6923d57c-6c82-4fe8-bd99-15e3be887ce1-kube-api-access-sgqk4\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b\" (UID: \"6923d57c-6c82-4fe8-bd99-15e3be887ce1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b"
Nov 03 08:15:00.405004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.397563    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qkpg\" (UniqueName: \"kubernetes.io/projected/6200af12-2c64-4113-b61b-e754c746c706-kube-api-access-4qkpg\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg\" (UID: \"6200af12-2c64-4113-b61b-e754c746c706\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg"
Nov 03 08:15:00.405004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.397719    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6200af12-2c64-4113-b61b-e754c746c706-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg\" (UID: \"6200af12-2c64-4113-b61b-e754c746c706\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg"
Nov 03 08:15:00.405004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.397749    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a05958e-20e4-4050-a9a8-76c22342d517-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg\" (UID: \"8a05958e-20e4-4050-a9a8-76c22342d517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg"
Nov 03 08:15:00.405004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.400743    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6086e8f-a1b6-43e2-8a92-3a962774083c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm\" (UID: \"e6086e8f-a1b6-43e2-8a92-3a962774083c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm"
Nov 03 08:15:00.405004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.400951    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6923d57c-6c82-4fe8-bd99-15e3be887ce1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b\" (UID: \"6923d57c-6c82-4fe8-bd99-15e3be887ce1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b"
Nov 03 08:15:00.405004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.400999    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt2rk\" (UniqueName: \"kubernetes.io/projected/8a05958e-20e4-4050-a9a8-76c22342d517-kube-api-access-pt2rk\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg\" (UID: \"8a05958e-20e4-4050-a9a8-76c22342d517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg"
Nov 03 08:15:00.405004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.401978    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a05958e-20e4-4050-a9a8-76c22342d517-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg\" (UID: \"8a05958e-20e4-4050-a9a8-76c22342d517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg"
Nov 03 08:15:00.405004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.402903    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6086e8f-a1b6-43e2-8a92-3a962774083c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm\" (UID: \"e6086e8f-a1b6-43e2-8a92-3a962774083c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm"
Nov 03 08:15:00.410054 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.409990    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6923d57c-6c82-4fe8-bd99-15e3be887ce1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b\" (UID: \"6923d57c-6c82-4fe8-bd99-15e3be887ce1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b"
Nov 03 08:15:00.412855 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.412353    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f"]
Nov 03 08:15:00.447531 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.431163    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sg8p\" (UniqueName: \"kubernetes.io/projected/e6086e8f-a1b6-43e2-8a92-3a962774083c-kube-api-access-9sg8p\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm\" (UID: \"e6086e8f-a1b6-43e2-8a92-3a962774083c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm"
Nov 03 08:15:00.447531 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.441783    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgqk4\" (UniqueName: \"kubernetes.io/projected/6923d57c-6c82-4fe8-bd99-15e3be887ce1-kube-api-access-sgqk4\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b\" (UID: \"6923d57c-6c82-4fe8-bd99-15e3be887ce1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b"
Nov 03 08:15:00.447531 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.445267    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt2rk\" (UniqueName: \"kubernetes.io/projected/8a05958e-20e4-4050-a9a8-76c22342d517-kube-api-access-pt2rk\") pod \"maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg\" (UID: \"8a05958e-20e4-4050-a9a8-76c22342d517\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg"
Nov 03 08:15:00.487907 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.485005    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv"]
Nov 03 08:15:00.490057 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.488914    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f"
Nov 03 08:15:00.491437 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.490955    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8"]
Nov 03 08:15:00.496910 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.493846    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv"
Nov 03 08:15:00.496910 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.495514    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg"
Nov 03 08:15:00.496910 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.496752    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd"]
Nov 03 08:15:00.496910 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.496807    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv"]
Nov 03 08:15:00.496910 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.496870    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8"
Nov 03 08:15:00.502043 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.501427    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f"]
Nov 03 08:15:00.502175 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.502052    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f228239e-f1d7-4bb8-9fc9-edc630a9df41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt\" (UID: \"f228239e-f1d7-4bb8-9fc9-edc630a9df41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt"
Nov 03 08:15:00.502175 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.502146    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qkpg\" (UniqueName: \"kubernetes.io/projected/6200af12-2c64-4113-b61b-e754c746c706-kube-api-access-4qkpg\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg\" (UID: \"6200af12-2c64-4113-b61b-e754c746c706\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg"
Nov 03 08:15:00.502326 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.502227    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6200af12-2c64-4113-b61b-e754c746c706-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg\" (UID: \"6200af12-2c64-4113-b61b-e754c746c706\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg"
Nov 03 08:15:00.502326 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.502246    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwsl9\" (UniqueName: \"kubernetes.io/projected/6f9a2a18-912c-4366-94b9-96770048ec4d-kube-api-access-vwsl9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd\" (UID: \"6f9a2a18-912c-4366-94b9-96770048ec4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd"
Nov 03 08:15:00.502458 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.502348    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9a2a18-912c-4366-94b9-96770048ec4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd\" (UID: \"6f9a2a18-912c-4366-94b9-96770048ec4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd"
Nov 03 08:15:00.502458 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.502452    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmv6j\" (UniqueName: \"kubernetes.io/projected/f228239e-f1d7-4bb8-9fc9-edc630a9df41-kube-api-access-vmv6j\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt\" (UID: \"f228239e-f1d7-4bb8-9fc9-edc630a9df41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt"
Nov 03 08:15:00.503417 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.503395    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f228239e-f1d7-4bb8-9fc9-edc630a9df41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt\" (UID: \"f228239e-f1d7-4bb8-9fc9-edc630a9df41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt"
Nov 03 08:15:00.503564 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.503395    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6200af12-2c64-4113-b61b-e754c746c706-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg\" (UID: \"6200af12-2c64-4113-b61b-e754c746c706\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg"
Nov 03 08:15:00.513957 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.513935    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm"]
Nov 03 08:15:00.519862 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.519799    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmv6j\" (UniqueName: \"kubernetes.io/projected/f228239e-f1d7-4bb8-9fc9-edc630a9df41-kube-api-access-vmv6j\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt\" (UID: \"f228239e-f1d7-4bb8-9fc9-edc630a9df41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt"
Nov 03 08:15:00.520523 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.520500    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qkpg\" (UniqueName: \"kubernetes.io/projected/6200af12-2c64-4113-b61b-e754c746c706-kube-api-access-4qkpg\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg\" (UID: \"6200af12-2c64-4113-b61b-e754c746c706\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg"
Nov 03 08:15:00.524481 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.524456    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk"]
Nov 03 08:15:00.531582 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.531561    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk"
Nov 03 08:15:00.533933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.533913    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz"]
Nov 03 08:15:00.539720 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.539447    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz"
Nov 03 08:15:00.544616 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.544571    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw"]
Nov 03 08:15:00.547692 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.547633    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b"
Nov 03 08:15:00.555641 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.554449    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8"]
Nov 03 08:15:00.555641 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.554528    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw"
Nov 03 08:15:00.565291 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.565265    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs"]
Nov 03 08:15:00.570387 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.570372    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs"
Nov 03 08:15:00.579143 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.579121    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk"]
Nov 03 08:15:00.581400 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.581377    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz"]
Nov 03 08:15:00.588795 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.586174    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs"]
Nov 03 08:15:00.606811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.603392    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw"]
Nov 03 08:15:00.606811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.604364    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a5d91d5-e0e2-41db-bea2-85a814e5274a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw\" (UID: \"5a5d91d5-e0e2-41db-bea2-85a814e5274a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw"
Nov 03 08:15:00.606811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.604429    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8237d5fa-c742-4293-802c-3fc5e29c52f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f\" (UID: \"8237d5fa-c742-4293-802c-3fc5e29c52f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f"
Nov 03 08:15:00.606811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.604469    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwsl9\" (UniqueName: \"kubernetes.io/projected/6f9a2a18-912c-4366-94b9-96770048ec4d-kube-api-access-vwsl9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd\" (UID: \"6f9a2a18-912c-4366-94b9-96770048ec4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd"
Nov 03 08:15:00.606811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.604503    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b9bn\" (UniqueName: \"kubernetes.io/projected/79d22235-c4cd-4701-a8ab-27f8143ce0fa-kube-api-access-4b9bn\") pod \"maas-api-key-cleanup-29894895-rn8xs\" (UID: \"79d22235-c4cd-4701-a8ab-27f8143ce0fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs"
Nov 03 08:15:00.606811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.604527    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6cb9d88-cc98-44a7-9e72-ace1b878668f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8\" (UID: \"b6cb9d88-cc98-44a7-9e72-ace1b878668f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8"
Nov 03 08:15:00.606811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.604597    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9a2a18-912c-4366-94b9-96770048ec4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd\" (UID: \"6f9a2a18-912c-4366-94b9-96770048ec4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd"
Nov 03 08:15:00.606811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.604640    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89nsf\" (UniqueName: \"kubernetes.io/projected/86e2d036-060e-40b6-8584-bd61b2595169-kube-api-access-89nsf\") pod \"maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv\" (UID: \"86e2d036-060e-40b6-8584-bd61b2595169\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv"
Nov 03 08:15:00.606811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.604657    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8gzq\" (UniqueName: \"kubernetes.io/projected/b6cb9d88-cc98-44a7-9e72-ace1b878668f-kube-api-access-b8gzq\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8\" (UID: \"b6cb9d88-cc98-44a7-9e72-ace1b878668f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8"
Nov 03 08:15:00.606811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.604686    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/250bea66-7bc7-41ba-b7a4-2bc0498ed885-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz\" (UID: \"250bea66-7bc7-41ba-b7a4-2bc0498ed885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz"
Nov 03 08:15:00.606811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.604707    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmvrg\" (UniqueName: \"kubernetes.io/projected/5a5d91d5-e0e2-41db-bea2-85a814e5274a-kube-api-access-jmvrg\") pod \"maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw\" (UID: \"5a5d91d5-e0e2-41db-bea2-85a814e5274a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw"
Nov 03 08:15:00.606811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.604817    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngjs5\" (UniqueName: \"kubernetes.io/projected/754211f3-c239-4469-be61-86b3bf06326c-kube-api-access-ngjs5\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk\" (UID: \"754211f3-c239-4469-be61-86b3bf06326c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk"
Nov 03 08:15:00.606811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.604871    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/754211f3-c239-4469-be61-86b3bf06326c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk\" (UID: \"754211f3-c239-4469-be61-86b3bf06326c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk"
Nov 03 08:15:00.606811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.604901    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6zhk\" (UniqueName: \"kubernetes.io/projected/250bea66-7bc7-41ba-b7a4-2bc0498ed885-kube-api-access-w6zhk\") pod \"maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz\" (UID: \"250bea66-7bc7-41ba-b7a4-2bc0498ed885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz"
Nov 03 08:15:00.606811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.604971    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86e2d036-060e-40b6-8584-bd61b2595169-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv\" (UID: \"86e2d036-060e-40b6-8584-bd61b2595169\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv"
Nov 03 08:15:00.606811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.605013    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6dnj\" (UniqueName: \"kubernetes.io/projected/8237d5fa-c742-4293-802c-3fc5e29c52f0-kube-api-access-m6dnj\") pod \"maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f\" (UID: \"8237d5fa-c742-4293-802c-3fc5e29c52f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f"
Nov 03 08:15:00.606811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.605063    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79d22235-c4cd-4701-a8ab-27f8143ce0fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894895-rn8xs\" (UID: \"79d22235-c4cd-4701-a8ab-27f8143ce0fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs"
Nov 03 08:15:00.607862 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.606861    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9a2a18-912c-4366-94b9-96770048ec4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd\" (UID: \"6f9a2a18-912c-4366-94b9-96770048ec4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd"
Nov 03 08:15:00.612076 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.611916    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm"
Nov 03 08:15:00.628741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.628683    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwsl9\" (UniqueName: \"kubernetes.io/projected/6f9a2a18-912c-4366-94b9-96770048ec4d-kube-api-access-vwsl9\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd\" (UID: \"6f9a2a18-912c-4366-94b9-96770048ec4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd"
Nov 03 08:15:00.706684 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.706656    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a5d91d5-e0e2-41db-bea2-85a814e5274a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw\" (UID: \"5a5d91d5-e0e2-41db-bea2-85a814e5274a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw"
Nov 03 08:15:00.706831 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.706731    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8237d5fa-c742-4293-802c-3fc5e29c52f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f\" (UID: \"8237d5fa-c742-4293-802c-3fc5e29c52f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f"
Nov 03 08:15:00.706831 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.706795    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b9bn\" (UniqueName: \"kubernetes.io/projected/79d22235-c4cd-4701-a8ab-27f8143ce0fa-kube-api-access-4b9bn\") pod \"maas-api-key-cleanup-29894895-rn8xs\" (UID: \"79d22235-c4cd-4701-a8ab-27f8143ce0fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs"
Nov 03 08:15:00.706831 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.706824    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6cb9d88-cc98-44a7-9e72-ace1b878668f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8\" (UID: \"b6cb9d88-cc98-44a7-9e72-ace1b878668f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8"
Nov 03 08:15:00.707004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.706959    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89nsf\" (UniqueName: \"kubernetes.io/projected/86e2d036-060e-40b6-8584-bd61b2595169-kube-api-access-89nsf\") pod \"maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv\" (UID: \"86e2d036-060e-40b6-8584-bd61b2595169\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv"
Nov 03 08:15:00.707004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.706981    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8gzq\" (UniqueName: \"kubernetes.io/projected/b6cb9d88-cc98-44a7-9e72-ace1b878668f-kube-api-access-b8gzq\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8\" (UID: \"b6cb9d88-cc98-44a7-9e72-ace1b878668f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8"
Nov 03 08:15:00.707105 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.707003    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/250bea66-7bc7-41ba-b7a4-2bc0498ed885-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz\" (UID: \"250bea66-7bc7-41ba-b7a4-2bc0498ed885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz"
Nov 03 08:15:00.707105 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.707025    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmvrg\" (UniqueName: \"kubernetes.io/projected/5a5d91d5-e0e2-41db-bea2-85a814e5274a-kube-api-access-jmvrg\") pod \"maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw\" (UID: \"5a5d91d5-e0e2-41db-bea2-85a814e5274a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw"
Nov 03 08:15:00.707105 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.707089    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngjs5\" (UniqueName: \"kubernetes.io/projected/754211f3-c239-4469-be61-86b3bf06326c-kube-api-access-ngjs5\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk\" (UID: \"754211f3-c239-4469-be61-86b3bf06326c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk"
Nov 03 08:15:00.707260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.707122    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/754211f3-c239-4469-be61-86b3bf06326c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk\" (UID: \"754211f3-c239-4469-be61-86b3bf06326c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk"
Nov 03 08:15:00.707260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.707138    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6zhk\" (UniqueName: \"kubernetes.io/projected/250bea66-7bc7-41ba-b7a4-2bc0498ed885-kube-api-access-w6zhk\") pod \"maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz\" (UID: \"250bea66-7bc7-41ba-b7a4-2bc0498ed885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz"
Nov 03 08:15:00.707260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.707183    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86e2d036-060e-40b6-8584-bd61b2595169-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv\" (UID: \"86e2d036-060e-40b6-8584-bd61b2595169\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv"
Nov 03 08:15:00.707260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.707208    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6dnj\" (UniqueName: \"kubernetes.io/projected/8237d5fa-c742-4293-802c-3fc5e29c52f0-kube-api-access-m6dnj\") pod \"maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f\" (UID: \"8237d5fa-c742-4293-802c-3fc5e29c52f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f"
Nov 03 08:15:00.707260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.707228    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79d22235-c4cd-4701-a8ab-27f8143ce0fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894895-rn8xs\" (UID: \"79d22235-c4cd-4701-a8ab-27f8143ce0fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs"
Nov 03 08:15:00.710438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.707868    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79d22235-c4cd-4701-a8ab-27f8143ce0fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894895-rn8xs\" (UID: \"79d22235-c4cd-4701-a8ab-27f8143ce0fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs"
Nov 03 08:15:00.710438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.708516    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a5d91d5-e0e2-41db-bea2-85a814e5274a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw\" (UID: \"5a5d91d5-e0e2-41db-bea2-85a814e5274a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw"
Nov 03 08:15:00.710438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.709006    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/754211f3-c239-4469-be61-86b3bf06326c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk\" (UID: \"754211f3-c239-4469-be61-86b3bf06326c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk"
Nov 03 08:15:00.710438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.709720    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86e2d036-060e-40b6-8584-bd61b2595169-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv\" (UID: \"86e2d036-060e-40b6-8584-bd61b2595169\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv"
Nov 03 08:15:00.710438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.710172    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6cb9d88-cc98-44a7-9e72-ace1b878668f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8\" (UID: \"b6cb9d88-cc98-44a7-9e72-ace1b878668f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8"
Nov 03 08:15:00.710438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.710324    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8237d5fa-c742-4293-802c-3fc5e29c52f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f\" (UID: \"8237d5fa-c742-4293-802c-3fc5e29c52f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f"
Nov 03 08:15:00.711011 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.710985    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/250bea66-7bc7-41ba-b7a4-2bc0498ed885-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz\" (UID: \"250bea66-7bc7-41ba-b7a4-2bc0498ed885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz"
Nov 03 08:15:00.722863 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.722464    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6dnj\" (UniqueName: \"kubernetes.io/projected/8237d5fa-c742-4293-802c-3fc5e29c52f0-kube-api-access-m6dnj\") pod \"maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f\" (UID: \"8237d5fa-c742-4293-802c-3fc5e29c52f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f"
Nov 03 08:15:00.727028 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.727004    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89nsf\" (UniqueName: \"kubernetes.io/projected/86e2d036-060e-40b6-8584-bd61b2595169-kube-api-access-89nsf\") pod \"maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv\" (UID: \"86e2d036-060e-40b6-8584-bd61b2595169\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv"
Nov 03 08:15:00.738194 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.738163    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmvrg\" (UniqueName: \"kubernetes.io/projected/5a5d91d5-e0e2-41db-bea2-85a814e5274a-kube-api-access-jmvrg\") pod \"maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw\" (UID: \"5a5d91d5-e0e2-41db-bea2-85a814e5274a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw"
Nov 03 08:15:00.739028 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.738187    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8gzq\" (UniqueName: \"kubernetes.io/projected/b6cb9d88-cc98-44a7-9e72-ace1b878668f-kube-api-access-b8gzq\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8\" (UID: \"b6cb9d88-cc98-44a7-9e72-ace1b878668f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8"
Nov 03 08:15:00.739028 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.738916    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngjs5\" (UniqueName: \"kubernetes.io/projected/754211f3-c239-4469-be61-86b3bf06326c-kube-api-access-ngjs5\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk\" (UID: \"754211f3-c239-4469-be61-86b3bf06326c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk"
Nov 03 08:15:00.740663 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.740634    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b9bn\" (UniqueName: \"kubernetes.io/projected/79d22235-c4cd-4701-a8ab-27f8143ce0fa-kube-api-access-4b9bn\") pod \"maas-api-key-cleanup-29894895-rn8xs\" (UID: \"79d22235-c4cd-4701-a8ab-27f8143ce0fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs"
Nov 03 08:15:00.741847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.741810    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6zhk\" (UniqueName: \"kubernetes.io/projected/250bea66-7bc7-41ba-b7a4-2bc0498ed885-kube-api-access-w6zhk\") pod \"maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz\" (UID: \"250bea66-7bc7-41ba-b7a4-2bc0498ed885\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz"
Nov 03 08:15:00.743831 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.743807    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs"
Nov 03 08:15:00.784927 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.784388    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg"
Nov 03 08:15:00.818096 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.818063    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt"
Nov 03 08:15:00.873834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.842253    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd"
Nov 03 08:15:00.873834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.869067    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f"
Nov 03 08:15:00.900937 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.900901    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg"]
Nov 03 08:15:00.919151 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:15:00.918988    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a05958e_20e4_4050_a9a8_76c22342d517.slice/crio-782cf33f3a9b1bbcb491d06f655a62ebc9e6befc2848c544e8c9cfae20a49672 WatchSource:0}: Error finding container 782cf33f3a9b1bbcb491d06f655a62ebc9e6befc2848c544e8c9cfae20a49672: Status 404 returned error can't find the container with id 782cf33f3a9b1bbcb491d06f655a62ebc9e6befc2848c544e8c9cfae20a49672
Nov 03 08:15:00.925350 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.924409    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv"
Nov 03 08:15:00.970049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.969581    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8"
Nov 03 08:15:00.992323 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:00.992091    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk"
Nov 03 08:15:01.025114 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:01.025082    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz"
Nov 03 08:15:01.034615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:01.034291    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw"
Nov 03 08:15:01.080358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:01.080222    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b"]
Nov 03 08:15:01.296409 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:01.296316    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm"]
Nov 03 08:15:01.771872 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:01.771747    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b" event={"ID":"6923d57c-6c82-4fe8-bd99-15e3be887ce1","Type":"ContainerStarted","Data":"334628b19ffe23d434a54f0791f9c92a7639f40463527baab9dd6259906ec895"}
Nov 03 08:15:01.783976 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:01.783945    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg" event={"ID":"8a05958e-20e4-4050-a9a8-76c22342d517","Type":"ContainerStarted","Data":"782cf33f3a9b1bbcb491d06f655a62ebc9e6befc2848c544e8c9cfae20a49672"}
Nov 03 08:15:01.837302 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:01.837274    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs"]
Nov 03 08:15:01.871639 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:01.871612    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f"]
Nov 03 08:15:01.873267 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:15:01.873243    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8237d5fa_c742_4293_802c_3fc5e29c52f0.slice/crio-3a99a13fb8326caa66213ea799e97afd0d8eafc8ee74d0304f845cf76c397af5 WatchSource:0}: Error finding container 3a99a13fb8326caa66213ea799e97afd0d8eafc8ee74d0304f845cf76c397af5: Status 404 returned error can't find the container with id 3a99a13fb8326caa66213ea799e97afd0d8eafc8ee74d0304f845cf76c397af5
Nov 03 08:15:01.929922 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:01.929839    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd"]
Nov 03 08:15:01.932947 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:15:01.932917    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f9a2a18_912c_4366_94b9_96770048ec4d.slice/crio-87be978c8ecacad2e19d064008fe95909037a41f6395a87d23227d75efe88ddd WatchSource:0}: Error finding container 87be978c8ecacad2e19d064008fe95909037a41f6395a87d23227d75efe88ddd: Status 404 returned error can't find the container with id 87be978c8ecacad2e19d064008fe95909037a41f6395a87d23227d75efe88ddd
Nov 03 08:15:02.081661 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.081581    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg"]
Nov 03 08:15:02.091232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.087149    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw"]
Nov 03 08:15:02.110212 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.110173    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8"]
Nov 03 08:15:02.147201 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.147130    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk"]
Nov 03 08:15:02.177223 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:15:02.177183    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a5d91d5_e0e2_41db_bea2_85a814e5274a.slice/crio-b86265e065da482028773256d21b071c09a239929035669664c08266fedb4fe7 WatchSource:0}: Error finding container b86265e065da482028773256d21b071c09a239929035669664c08266fedb4fe7: Status 404 returned error can't find the container with id b86265e065da482028773256d21b071c09a239929035669664c08266fedb4fe7
Nov 03 08:15:02.179594 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:15:02.179043    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6200af12_2c64_4113_b61b_e754c746c706.slice/crio-b4c10b6160fde13c21dc318dcd8289bea5dde126a8b9ca3be770066806609305 WatchSource:0}: Error finding container b4c10b6160fde13c21dc318dcd8289bea5dde126a8b9ca3be770066806609305: Status 404 returned error can't find the container with id b4c10b6160fde13c21dc318dcd8289bea5dde126a8b9ca3be770066806609305
Nov 03 08:15:02.196996 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.196975    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz"]
Nov 03 08:15:02.208102 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.208081    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt"]
Nov 03 08:15:02.212422 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.212399    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv"]
Nov 03 08:15:02.258735 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:15:02.258711    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6cb9d88_cc98_44a7_9e72_ace1b878668f.slice/crio-ee67c247fe2c454ce07a96993a0c15b098e6451cb33f72de3eace60c7bae4ad4 WatchSource:0}: Error finding container ee67c247fe2c454ce07a96993a0c15b098e6451cb33f72de3eace60c7bae4ad4: Status 404 returned error can't find the container with id ee67c247fe2c454ce07a96993a0c15b098e6451cb33f72de3eace60c7bae4ad4
Nov 03 08:15:02.260550 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:15:02.260528    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod754211f3_c239_4469_be61_86b3bf06326c.slice/crio-582f3fc3ea2b696908b60b496184ca7d33c179f293d4244f71d8005e2c2e6e7e WatchSource:0}: Error finding container 582f3fc3ea2b696908b60b496184ca7d33c179f293d4244f71d8005e2c2e6e7e: Status 404 returned error can't find the container with id 582f3fc3ea2b696908b60b496184ca7d33c179f293d4244f71d8005e2c2e6e7e
Nov 03 08:15:02.261846 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:15:02.261821    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250bea66_7bc7_41ba_b7a4_2bc0498ed885.slice/crio-e18eed1fc752b4b0fc8e50b15fa9710101e7ff19224789207583386a877f3de5 WatchSource:0}: Error finding container e18eed1fc752b4b0fc8e50b15fa9710101e7ff19224789207583386a877f3de5: Status 404 returned error can't find the container with id e18eed1fc752b4b0fc8e50b15fa9710101e7ff19224789207583386a877f3de5
Nov 03 08:15:02.360904 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:15:02.360877    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf228239e_f1d7_4bb8_9fc9_edc630a9df41.slice/crio-40ad98561d7e55ebeda13e6d4ce718706de5d07cf25984186857b3e607a33b15 WatchSource:0}: Error finding container 40ad98561d7e55ebeda13e6d4ce718706de5d07cf25984186857b3e607a33b15: Status 404 returned error can't find the container with id 40ad98561d7e55ebeda13e6d4ce718706de5d07cf25984186857b3e607a33b15
Nov 03 08:15:02.799853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.799822    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b" event={"ID":"6923d57c-6c82-4fe8-bd99-15e3be887ce1","Type":"ContainerStarted","Data":"66565fb0b62ce496a028f2d6b3aa31fc6c084192a0e7c2a0adef2c46e1278de9"}
Nov 03 08:15:02.802068 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.802039    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz" event={"ID":"250bea66-7bc7-41ba-b7a4-2bc0498ed885","Type":"ContainerStarted","Data":"e18eed1fc752b4b0fc8e50b15fa9710101e7ff19224789207583386a877f3de5"}
Nov 03 08:15:02.803274 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.803248    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt" event={"ID":"f228239e-f1d7-4bb8-9fc9-edc630a9df41","Type":"ContainerStarted","Data":"40ad98561d7e55ebeda13e6d4ce718706de5d07cf25984186857b3e607a33b15"}
Nov 03 08:15:02.804688 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.804667    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs" event={"ID":"79d22235-c4cd-4701-a8ab-27f8143ce0fa","Type":"ContainerStarted","Data":"c0c42b9f22449995de298833eb344d21b516fc17a3c5d0b66ac98c16483a528e"}
Nov 03 08:15:02.804860 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.804706    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs" event={"ID":"79d22235-c4cd-4701-a8ab-27f8143ce0fa","Type":"ContainerStarted","Data":"5d0f80cd393dc1e0d7659d9b2df7b18ca5e71a39a8ab6d5590957a0a496db5af"}
Nov 03 08:15:02.805920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.805898    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8" event={"ID":"b6cb9d88-cc98-44a7-9e72-ace1b878668f","Type":"ContainerStarted","Data":"ee67c247fe2c454ce07a96993a0c15b098e6451cb33f72de3eace60c7bae4ad4"}
Nov 03 08:15:02.807352 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.807323    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd" event={"ID":"6f9a2a18-912c-4366-94b9-96770048ec4d","Type":"ContainerStarted","Data":"d25fad067e21fd7a63e2b268f59b9d8d72fc2090fb2cd0f0c515dae26984fc8e"}
Nov 03 08:15:02.807457 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.807368    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd" event={"ID":"6f9a2a18-912c-4366-94b9-96770048ec4d","Type":"ContainerStarted","Data":"87be978c8ecacad2e19d064008fe95909037a41f6395a87d23227d75efe88ddd"}
Nov 03 08:15:02.808823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.808802    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f" event={"ID":"8237d5fa-c742-4293-802c-3fc5e29c52f0","Type":"ContainerStarted","Data":"81b601a0e2b03d773671c1d7a7368b499085d38a20f22daf32b058ea68c40f99"}
Nov 03 08:15:02.808906 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.808845    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f" event={"ID":"8237d5fa-c742-4293-802c-3fc5e29c52f0","Type":"ContainerStarted","Data":"3a99a13fb8326caa66213ea799e97afd0d8eafc8ee74d0304f845cf76c397af5"}
Nov 03 08:15:02.810015 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.809996    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw" event={"ID":"5a5d91d5-e0e2-41db-bea2-85a814e5274a","Type":"ContainerStarted","Data":"b86265e065da482028773256d21b071c09a239929035669664c08266fedb4fe7"}
Nov 03 08:15:02.811160 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.811139    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk" event={"ID":"754211f3-c239-4469-be61-86b3bf06326c","Type":"ContainerStarted","Data":"582f3fc3ea2b696908b60b496184ca7d33c179f293d4244f71d8005e2c2e6e7e"}
Nov 03 08:15:02.812595 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.812570    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm" event={"ID":"e6086e8f-a1b6-43e2-8a92-3a962774083c","Type":"ContainerStarted","Data":"cedf9ed1e00e369f08570d9c7d811578b59a9d5cbb440b8394a1a3ee509e428a"}
Nov 03 08:15:02.812679 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.812611    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm" event={"ID":"e6086e8f-a1b6-43e2-8a92-3a962774083c","Type":"ContainerStarted","Data":"3e142f2d35ad8dd7b29ba715e935b89880b48a8ab23ef96b28a14166fa448c1a"}
Nov 03 08:15:02.814086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.814058    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv" event={"ID":"86e2d036-060e-40b6-8584-bd61b2595169","Type":"ContainerStarted","Data":"e0cdc3ef9810725602a09af8d6e65d029be36a6080668c3e9ac7bdab408b68e0"}
Nov 03 08:15:02.815206 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.815163    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b" podStartSLOduration=2.8151503509999998 podStartE2EDuration="2.815150351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:02.812728646 +0000 UTC m=+1300.927192579" watchObservedRunningTime="2026-11-03 08:15:02.815150351 +0000 UTC m=+1300.929614326"
Nov 03 08:15:02.815425 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.815404    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg" event={"ID":"6200af12-2c64-4113-b61b-e754c746c706","Type":"ContainerStarted","Data":"b4c10b6160fde13c21dc318dcd8289bea5dde126a8b9ca3be770066806609305"}
Nov 03 08:15:02.817239 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.817219    5070 generic.go:356] "Generic (PLEG): container finished" podID="8a05958e-20e4-4050-a9a8-76c22342d517" containerID="29d7cc200671290368e56e49366c1dcee92b1e6327396f339e7ce1458b79a3fc" exitCode=60
Nov 03 08:15:02.817339 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.817257    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg" event={"ID":"8a05958e-20e4-4050-a9a8-76c22342d517","Type":"ContainerDied","Data":"29d7cc200671290368e56e49366c1dcee92b1e6327396f339e7ce1458b79a3fc"}
Nov 03 08:15:02.817542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.817526    5070 scope.go:122] "RemoveContainer" containerID="29d7cc200671290368e56e49366c1dcee92b1e6327396f339e7ce1458b79a3fc"
Nov 03 08:15:02.834612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.834576    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm" podStartSLOduration=2.834564711 podStartE2EDuration="2.834564711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:02.826885659 +0000 UTC m=+1300.941349627" watchObservedRunningTime="2026-11-03 08:15:02.834564711 +0000 UTC m=+1300.949028642"
Nov 03 08:15:02.841029 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.840964    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f" podStartSLOduration=2.840949716 podStartE2EDuration="2.840949716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:02.840136379 +0000 UTC m=+1300.954600306" watchObservedRunningTime="2026-11-03 08:15:02.840949716 +0000 UTC m=+1300.955413642"
Nov 03 08:15:02.856501 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.856467    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs" podStartSLOduration=2.856457028 podStartE2EDuration="2.856457028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:02.853611722 +0000 UTC m=+1300.968075653" watchObservedRunningTime="2026-11-03 08:15:02.856457028 +0000 UTC m=+1300.970920959"
Nov 03 08:15:02.944220 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:02.939681    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd" podStartSLOduration=2.939660977 podStartE2EDuration="2.939660977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:02.866383447 +0000 UTC m=+1300.980847379" watchObservedRunningTime="2026-11-03 08:15:02.939660977 +0000 UTC m=+1301.054124909"
Nov 03 08:15:03.082883 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:03.082851    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77803_4112_45fe_bf9b_5ca069a5441f.slice/crio-71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77803_4112_45fe_bf9b_5ca069a5441f.slice/crio-conmon-71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 08:15:03.828470 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.828394    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv" event={"ID":"86e2d036-060e-40b6-8584-bd61b2595169","Type":"ContainerStarted","Data":"8b72174647f870b6106cbec92778f9b4cf22bc9236a14fe64c2e934ef11e8074"}
Nov 03 08:15:03.830350 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.830322    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg" event={"ID":"6200af12-2c64-4113-b61b-e754c746c706","Type":"ContainerStarted","Data":"9aa2980154ed0252abd894834d5dfab1a859fef3ecf761ba1e35a2945e3ff65e"}
Nov 03 08:15:03.832359 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.832339    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg" event={"ID":"8a05958e-20e4-4050-a9a8-76c22342d517","Type":"ContainerStarted","Data":"7b9d84c721107532f48e0d0e796aa280d6c87b21747fe641d297dbad00085568"}
Nov 03 08:15:03.834144 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.834117    5070 generic.go:356] "Generic (PLEG): container finished" podID="6923d57c-6c82-4fe8-bd99-15e3be887ce1" containerID="66565fb0b62ce496a028f2d6b3aa31fc6c084192a0e7c2a0adef2c46e1278de9" exitCode=60
Nov 03 08:15:03.834233 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.834171    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b" event={"ID":"6923d57c-6c82-4fe8-bd99-15e3be887ce1","Type":"ContainerDied","Data":"66565fb0b62ce496a028f2d6b3aa31fc6c084192a0e7c2a0adef2c46e1278de9"}
Nov 03 08:15:03.834426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.834405    5070 scope.go:122] "RemoveContainer" containerID="66565fb0b62ce496a028f2d6b3aa31fc6c084192a0e7c2a0adef2c46e1278de9"
Nov 03 08:15:03.836225 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.836187    5070 generic.go:356] "Generic (PLEG): container finished" podID="250bea66-7bc7-41ba-b7a4-2bc0498ed885" containerID="907f6f53db6ffb773b90abfe80d0cb19348ad27bad19139be104b1ca274c3c4b" exitCode=60
Nov 03 08:15:03.836335 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.836317    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz" event={"ID":"250bea66-7bc7-41ba-b7a4-2bc0498ed885","Type":"ContainerDied","Data":"907f6f53db6ffb773b90abfe80d0cb19348ad27bad19139be104b1ca274c3c4b"}
Nov 03 08:15:03.836534 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.836515    5070 scope.go:122] "RemoveContainer" containerID="907f6f53db6ffb773b90abfe80d0cb19348ad27bad19139be104b1ca274c3c4b"
Nov 03 08:15:03.841083 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.841061    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt" event={"ID":"f228239e-f1d7-4bb8-9fc9-edc630a9df41","Type":"ContainerStarted","Data":"7685c74b17032b94394cac6d52c8c287b829b6cb6bc26599f020f18a37ad0732"}
Nov 03 08:15:03.842654 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.842634    5070 generic.go:356] "Generic (PLEG): container finished" podID="79d22235-c4cd-4701-a8ab-27f8143ce0fa" containerID="c0c42b9f22449995de298833eb344d21b516fc17a3c5d0b66ac98c16483a528e" exitCode=60
Nov 03 08:15:03.842741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.842723    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs" event={"ID":"79d22235-c4cd-4701-a8ab-27f8143ce0fa","Type":"ContainerDied","Data":"c0c42b9f22449995de298833eb344d21b516fc17a3c5d0b66ac98c16483a528e"}
Nov 03 08:15:03.842974 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.842960    5070 scope.go:122] "RemoveContainer" containerID="c0c42b9f22449995de298833eb344d21b516fc17a3c5d0b66ac98c16483a528e"
Nov 03 08:15:03.844969 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.844945    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8" event={"ID":"b6cb9d88-cc98-44a7-9e72-ace1b878668f","Type":"ContainerStarted","Data":"38e99a89fcd9bb63bd153a961d07b807c86f3d8b5010a1e23e9fc220c5ee0347"}
Nov 03 08:15:03.847365 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.847133    5070 scope.go:122] "RemoveContainer" containerID="d25fad067e21fd7a63e2b268f59b9d8d72fc2090fb2cd0f0c515dae26984fc8e"
Nov 03 08:15:03.847365 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.847143    5070 generic.go:356] "Generic (PLEG): container finished" podID="6f9a2a18-912c-4366-94b9-96770048ec4d" containerID="d25fad067e21fd7a63e2b268f59b9d8d72fc2090fb2cd0f0c515dae26984fc8e" exitCode=60
Nov 03 08:15:03.847365 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.847243    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd" event={"ID":"6f9a2a18-912c-4366-94b9-96770048ec4d","Type":"ContainerDied","Data":"d25fad067e21fd7a63e2b268f59b9d8d72fc2090fb2cd0f0c515dae26984fc8e"}
Nov 03 08:15:03.851472 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.851432    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv" podStartSLOduration=3.851416328 podStartE2EDuration="3.851416328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:03.846513466 +0000 UTC m=+1301.960977398" watchObservedRunningTime="2026-11-03 08:15:03.851416328 +0000 UTC m=+1301.965880259"
Nov 03 08:15:03.862927 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.861052    5070 generic.go:356] "Generic (PLEG): container finished" podID="8237d5fa-c742-4293-802c-3fc5e29c52f0" containerID="81b601a0e2b03d773671c1d7a7368b499085d38a20f22daf32b058ea68c40f99" exitCode=60
Nov 03 08:15:03.862927 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.861364    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f" event={"ID":"8237d5fa-c742-4293-802c-3fc5e29c52f0","Type":"ContainerDied","Data":"81b601a0e2b03d773671c1d7a7368b499085d38a20f22daf32b058ea68c40f99"}
Nov 03 08:15:03.862927 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.861616    5070 scope.go:122] "RemoveContainer" containerID="81b601a0e2b03d773671c1d7a7368b499085d38a20f22daf32b058ea68c40f99"
Nov 03 08:15:03.869111 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.869093    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw" event={"ID":"5a5d91d5-e0e2-41db-bea2-85a814e5274a","Type":"ContainerStarted","Data":"b5c6f8522ca7c08b665035ae86ca9d9e0da9c4b47654fc0aa52985fa0e98696c"}
Nov 03 08:15:03.869620 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.869585    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8" podStartSLOduration=3.869573789 podStartE2EDuration="3.869573789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:03.867185482 +0000 UTC m=+1301.981649414" watchObservedRunningTime="2026-11-03 08:15:03.869573789 +0000 UTC m=+1301.984037755"
Nov 03 08:15:03.873395 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.872075    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk" event={"ID":"754211f3-c239-4469-be61-86b3bf06326c","Type":"ContainerStarted","Data":"865faf182c801b9554ebf29000ef977068e5878a858f39ed7258826f8e18108f"}
Nov 03 08:15:03.889221 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.889198    5070 generic.go:356] "Generic (PLEG): container finished" podID="e6086e8f-a1b6-43e2-8a92-3a962774083c" containerID="cedf9ed1e00e369f08570d9c7d811578b59a9d5cbb440b8394a1a3ee509e428a" exitCode=60
Nov 03 08:15:03.889357 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.889340    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm" event={"ID":"e6086e8f-a1b6-43e2-8a92-3a962774083c","Type":"ContainerDied","Data":"cedf9ed1e00e369f08570d9c7d811578b59a9d5cbb440b8394a1a3ee509e428a"}
Nov 03 08:15:03.889749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.889730    5070 scope.go:122] "RemoveContainer" containerID="cedf9ed1e00e369f08570d9c7d811578b59a9d5cbb440b8394a1a3ee509e428a"
Nov 03 08:15:03.940094 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.940052    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt" podStartSLOduration=3.940039774 podStartE2EDuration="3.940039774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:03.934362957 +0000 UTC m=+1302.048826908" watchObservedRunningTime="2026-11-03 08:15:03.940039774 +0000 UTC m=+1302.054503704"
Nov 03 08:15:03.953800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.953712    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg" podStartSLOduration=3.95369658 podStartE2EDuration="3.95369658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:03.949953983 +0000 UTC m=+1302.064417927" watchObservedRunningTime="2026-11-03 08:15:03.95369658 +0000 UTC m=+1302.068160512"
Nov 03 08:15:03.969285 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:03.969245    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg" podStartSLOduration=3.969232162 podStartE2EDuration="3.969232162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:03.962706136 +0000 UTC m=+1302.077170067" watchObservedRunningTime="2026-11-03 08:15:03.969232162 +0000 UTC m=+1302.083696092"
Nov 03 08:15:04.012367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.012272    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw" podStartSLOduration=4.012254242 podStartE2EDuration="4.012254242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:04.005181015 +0000 UTC m=+1302.119644946" watchObservedRunningTime="2026-11-03 08:15:04.012254242 +0000 UTC m=+1302.126718173"
Nov 03 08:15:04.059394 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.057052    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk" podStartSLOduration=4.057032505 podStartE2EDuration="4.057032505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:04.021374255 +0000 UTC m=+1302.135838236" watchObservedRunningTime="2026-11-03 08:15:04.057032505 +0000 UTC m=+1302.171496437"
Nov 03 08:15:04.909196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.909161    5070 generic.go:356] "Generic (PLEG): container finished" podID="f228239e-f1d7-4bb8-9fc9-edc630a9df41" containerID="7685c74b17032b94394cac6d52c8c287b829b6cb6bc26599f020f18a37ad0732" exitCode=60
Nov 03 08:15:04.909899 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.909241    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt" event={"ID":"f228239e-f1d7-4bb8-9fc9-edc630a9df41","Type":"ContainerDied","Data":"7685c74b17032b94394cac6d52c8c287b829b6cb6bc26599f020f18a37ad0732"}
Nov 03 08:15:04.909899 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.909491    5070 scope.go:122] "RemoveContainer" containerID="7685c74b17032b94394cac6d52c8c287b829b6cb6bc26599f020f18a37ad0732"
Nov 03 08:15:04.911678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.911657    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs" event={"ID":"79d22235-c4cd-4701-a8ab-27f8143ce0fa","Type":"ContainerStarted","Data":"9f3eb208f7e3af33df0b83c10f0d4e488f5116159463051ebcb21a0b0dab4c89"}
Nov 03 08:15:04.913839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.913813    5070 generic.go:356] "Generic (PLEG): container finished" podID="b6cb9d88-cc98-44a7-9e72-ace1b878668f" containerID="38e99a89fcd9bb63bd153a961d07b807c86f3d8b5010a1e23e9fc220c5ee0347" exitCode=60
Nov 03 08:15:04.913955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.913894    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8" event={"ID":"b6cb9d88-cc98-44a7-9e72-ace1b878668f","Type":"ContainerDied","Data":"38e99a89fcd9bb63bd153a961d07b807c86f3d8b5010a1e23e9fc220c5ee0347"}
Nov 03 08:15:04.914128 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.914109    5070 scope.go:122] "RemoveContainer" containerID="38e99a89fcd9bb63bd153a961d07b807c86f3d8b5010a1e23e9fc220c5ee0347"
Nov 03 08:15:04.916086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.916063    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd" event={"ID":"6f9a2a18-912c-4366-94b9-96770048ec4d","Type":"ContainerStarted","Data":"dc81580473c252a040840e7b71efcb6388950c83a1c5becdfcdd8810765244bd"}
Nov 03 08:15:04.918005 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.917981    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f" event={"ID":"8237d5fa-c742-4293-802c-3fc5e29c52f0","Type":"ContainerStarted","Data":"3d2fe88b7b5bc5ce8220968c3edf3dfd9041f5c8085a02a6aeff987c76afa8f1"}
Nov 03 08:15:04.919682 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.919661    5070 generic.go:356] "Generic (PLEG): container finished" podID="5a5d91d5-e0e2-41db-bea2-85a814e5274a" containerID="b5c6f8522ca7c08b665035ae86ca9d9e0da9c4b47654fc0aa52985fa0e98696c" exitCode=60
Nov 03 08:15:04.919817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.919734    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw" event={"ID":"5a5d91d5-e0e2-41db-bea2-85a814e5274a","Type":"ContainerDied","Data":"b5c6f8522ca7c08b665035ae86ca9d9e0da9c4b47654fc0aa52985fa0e98696c"}
Nov 03 08:15:04.920548 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.919980    5070 scope.go:122] "RemoveContainer" containerID="b5c6f8522ca7c08b665035ae86ca9d9e0da9c4b47654fc0aa52985fa0e98696c"
Nov 03 08:15:04.921607 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.921588    5070 generic.go:356] "Generic (PLEG): container finished" podID="754211f3-c239-4469-be61-86b3bf06326c" containerID="865faf182c801b9554ebf29000ef977068e5878a858f39ed7258826f8e18108f" exitCode=60
Nov 03 08:15:04.921705 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.921671    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk" event={"ID":"754211f3-c239-4469-be61-86b3bf06326c","Type":"ContainerDied","Data":"865faf182c801b9554ebf29000ef977068e5878a858f39ed7258826f8e18108f"}
Nov 03 08:15:04.921942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.921925    5070 scope.go:122] "RemoveContainer" containerID="865faf182c801b9554ebf29000ef977068e5878a858f39ed7258826f8e18108f"
Nov 03 08:15:04.924194 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.923984    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm" event={"ID":"e6086e8f-a1b6-43e2-8a92-3a962774083c","Type":"ContainerStarted","Data":"c890d0b64b747239bf8f59791c408b4e8349527e5eefbac2babf5ebe027e8d6f"}
Nov 03 08:15:04.926281 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.926253    5070 generic.go:356] "Generic (PLEG): container finished" podID="86e2d036-060e-40b6-8584-bd61b2595169" containerID="8b72174647f870b6106cbec92778f9b4cf22bc9236a14fe64c2e934ef11e8074" exitCode=60
Nov 03 08:15:04.926366 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.926320    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv" event={"ID":"86e2d036-060e-40b6-8584-bd61b2595169","Type":"ContainerDied","Data":"8b72174647f870b6106cbec92778f9b4cf22bc9236a14fe64c2e934ef11e8074"}
Nov 03 08:15:04.926816 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.926639    5070 scope.go:122] "RemoveContainer" containerID="8b72174647f870b6106cbec92778f9b4cf22bc9236a14fe64c2e934ef11e8074"
Nov 03 08:15:04.928968 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.928947    5070 generic.go:356] "Generic (PLEG): container finished" podID="6200af12-2c64-4113-b61b-e754c746c706" containerID="9aa2980154ed0252abd894834d5dfab1a859fef3ecf761ba1e35a2945e3ff65e" exitCode=60
Nov 03 08:15:04.929067 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.929018    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg" event={"ID":"6200af12-2c64-4113-b61b-e754c746c706","Type":"ContainerDied","Data":"9aa2980154ed0252abd894834d5dfab1a859fef3ecf761ba1e35a2945e3ff65e"}
Nov 03 08:15:04.929213 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.929199    5070 scope.go:122] "RemoveContainer" containerID="9aa2980154ed0252abd894834d5dfab1a859fef3ecf761ba1e35a2945e3ff65e"
Nov 03 08:15:04.931493 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.931470    5070 generic.go:356] "Generic (PLEG): container finished" podID="8a05958e-20e4-4050-a9a8-76c22342d517" containerID="7b9d84c721107532f48e0d0e796aa280d6c87b21747fe641d297dbad00085568" exitCode=60
Nov 03 08:15:04.931597 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.931571    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg" event={"ID":"8a05958e-20e4-4050-a9a8-76c22342d517","Type":"ContainerDied","Data":"7b9d84c721107532f48e0d0e796aa280d6c87b21747fe641d297dbad00085568"}
Nov 03 08:15:04.931651 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.931613    5070 scope.go:122] "RemoveContainer" containerID="29d7cc200671290368e56e49366c1dcee92b1e6327396f339e7ce1458b79a3fc"
Nov 03 08:15:04.931849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.931819    5070 scope.go:122] "RemoveContainer" containerID="7b9d84c721107532f48e0d0e796aa280d6c87b21747fe641d297dbad00085568"
Nov 03 08:15:04.932116 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:04.932089    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg_odh-ai-gateway-infra(8a05958e-20e4-4050-a9a8-76c22342d517)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg" podUID="8a05958e-20e4-4050-a9a8-76c22342d517"
Nov 03 08:15:04.933809 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.933785    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b" event={"ID":"6923d57c-6c82-4fe8-bd99-15e3be887ce1","Type":"ContainerStarted","Data":"03d12b6bad11846510b86946b63e5647fa3e0064233da4b6371820d9167592e0"}
Nov 03 08:15:04.936066 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:04.936047    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz" event={"ID":"250bea66-7bc7-41ba-b7a4-2bc0498ed885","Type":"ContainerStarted","Data":"41a84167d752eb6100f2b95c7230b5d58a8f2c7e49108de55dffdba249831688"}
Nov 03 08:15:05.092306 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.092224    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz" podStartSLOduration=5.092211773 podStartE2EDuration="5.092211773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:05.091853129 +0000 UTC m=+1303.206317061" watchObservedRunningTime="2026-11-03 08:15:05.092211773 +0000 UTC m=+1303.206675703"
Nov 03 08:15:05.947384 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.947309    5070 generic.go:356] "Generic (PLEG): container finished" podID="250bea66-7bc7-41ba-b7a4-2bc0498ed885" containerID="41a84167d752eb6100f2b95c7230b5d58a8f2c7e49108de55dffdba249831688" exitCode=60
Nov 03 08:15:05.947810 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.947399    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz" event={"ID":"250bea66-7bc7-41ba-b7a4-2bc0498ed885","Type":"ContainerDied","Data":"41a84167d752eb6100f2b95c7230b5d58a8f2c7e49108de55dffdba249831688"}
Nov 03 08:15:05.947810 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.947475    5070 scope.go:122] "RemoveContainer" containerID="907f6f53db6ffb773b90abfe80d0cb19348ad27bad19139be104b1ca274c3c4b"
Nov 03 08:15:05.947810 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.947700    5070 scope.go:122] "RemoveContainer" containerID="41a84167d752eb6100f2b95c7230b5d58a8f2c7e49108de55dffdba249831688"
Nov 03 08:15:05.948096 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:05.948064    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz_odh-ai-gateway-infra(250bea66-7bc7-41ba-b7a4-2bc0498ed885)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz" podUID="250bea66-7bc7-41ba-b7a4-2bc0498ed885"
Nov 03 08:15:05.949811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.949786    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt" event={"ID":"f228239e-f1d7-4bb8-9fc9-edc630a9df41","Type":"ContainerStarted","Data":"8a4850e9a297800fcefff77ad5193de236fd851949e32e59bb9239ed9885daf9"}
Nov 03 08:15:05.951773 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.951740    5070 generic.go:356] "Generic (PLEG): container finished" podID="79d22235-c4cd-4701-a8ab-27f8143ce0fa" containerID="9f3eb208f7e3af33df0b83c10f0d4e488f5116159463051ebcb21a0b0dab4c89" exitCode=60
Nov 03 08:15:05.951888 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.951786    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs" event={"ID":"79d22235-c4cd-4701-a8ab-27f8143ce0fa","Type":"ContainerDied","Data":"9f3eb208f7e3af33df0b83c10f0d4e488f5116159463051ebcb21a0b0dab4c89"}
Nov 03 08:15:05.952068 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.952049    5070 scope.go:122] "RemoveContainer" containerID="9f3eb208f7e3af33df0b83c10f0d4e488f5116159463051ebcb21a0b0dab4c89"
Nov 03 08:15:05.952376 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:05.952357    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894895-rn8xs_odh-ai-gateway-infra(79d22235-c4cd-4701-a8ab-27f8143ce0fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs" podUID="79d22235-c4cd-4701-a8ab-27f8143ce0fa"
Nov 03 08:15:05.954217 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.954194    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8" event={"ID":"b6cb9d88-cc98-44a7-9e72-ace1b878668f","Type":"ContainerStarted","Data":"c8075c3010e861dde721481f5ffcbfb15dfe3531901610bc9288964cf7eba8e0"}
Nov 03 08:15:05.956736 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.956712    5070 generic.go:356] "Generic (PLEG): container finished" podID="6f9a2a18-912c-4366-94b9-96770048ec4d" containerID="dc81580473c252a040840e7b71efcb6388950c83a1c5becdfcdd8810765244bd" exitCode=60
Nov 03 08:15:05.957006 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.956958    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd" event={"ID":"6f9a2a18-912c-4366-94b9-96770048ec4d","Type":"ContainerDied","Data":"dc81580473c252a040840e7b71efcb6388950c83a1c5becdfcdd8810765244bd"}
Nov 03 08:15:05.957110 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.957092    5070 scope.go:122] "RemoveContainer" containerID="dc81580473c252a040840e7b71efcb6388950c83a1c5becdfcdd8810765244bd"
Nov 03 08:15:05.957413 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:05.957387    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd_odh-ai-gateway-infra(6f9a2a18-912c-4366-94b9-96770048ec4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd" podUID="6f9a2a18-912c-4366-94b9-96770048ec4d"
Nov 03 08:15:05.959712 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.959686    5070 generic.go:356] "Generic (PLEG): container finished" podID="8237d5fa-c742-4293-802c-3fc5e29c52f0" containerID="3d2fe88b7b5bc5ce8220968c3edf3dfd9041f5c8085a02a6aeff987c76afa8f1" exitCode=60
Nov 03 08:15:05.959845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.959815    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f" event={"ID":"8237d5fa-c742-4293-802c-3fc5e29c52f0","Type":"ContainerDied","Data":"3d2fe88b7b5bc5ce8220968c3edf3dfd9041f5c8085a02a6aeff987c76afa8f1"}
Nov 03 08:15:05.960158 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.960138    5070 scope.go:122] "RemoveContainer" containerID="3d2fe88b7b5bc5ce8220968c3edf3dfd9041f5c8085a02a6aeff987c76afa8f1"
Nov 03 08:15:05.960444 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:05.960419    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f_odh-ai-gateway-infra(8237d5fa-c742-4293-802c-3fc5e29c52f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f" podUID="8237d5fa-c742-4293-802c-3fc5e29c52f0"
Nov 03 08:15:05.963523 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.963502    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw" event={"ID":"5a5d91d5-e0e2-41db-bea2-85a814e5274a","Type":"ContainerStarted","Data":"286100bfd9be06845d0d45f5bc2f54f576ab1f3ef10a12d6c0c0c53bcf0a578d"}
Nov 03 08:15:05.965836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.965820    5070 scope.go:122] "RemoveContainer" containerID="c0c42b9f22449995de298833eb344d21b516fc17a3c5d0b66ac98c16483a528e"
Nov 03 08:15:05.966242 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.966224    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk" event={"ID":"754211f3-c239-4469-be61-86b3bf06326c","Type":"ContainerStarted","Data":"53a63b2b0b2df07aa52a6e9e97bb34e213531c7245f6f01f11fa89d1a7216310"}
Nov 03 08:15:05.968241 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.968223    5070 generic.go:356] "Generic (PLEG): container finished" podID="e6086e8f-a1b6-43e2-8a92-3a962774083c" containerID="c890d0b64b747239bf8f59791c408b4e8349527e5eefbac2babf5ebe027e8d6f" exitCode=60
Nov 03 08:15:05.968315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.968293    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm" event={"ID":"e6086e8f-a1b6-43e2-8a92-3a962774083c","Type":"ContainerDied","Data":"c890d0b64b747239bf8f59791c408b4e8349527e5eefbac2babf5ebe027e8d6f"}
Nov 03 08:15:05.968473 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.968457    5070 scope.go:122] "RemoveContainer" containerID="c890d0b64b747239bf8f59791c408b4e8349527e5eefbac2babf5ebe027e8d6f"
Nov 03 08:15:05.968701 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:05.968683    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm_odh-ai-gateway-infra(e6086e8f-a1b6-43e2-8a92-3a962774083c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm" podUID="e6086e8f-a1b6-43e2-8a92-3a962774083c"
Nov 03 08:15:05.971051 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.971023    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv" event={"ID":"86e2d036-060e-40b6-8584-bd61b2595169","Type":"ContainerStarted","Data":"ef8cba7a36936b014992ddd694c46a48a9d644f9ea00309ca04a86099a9ba9ac"}
Nov 03 08:15:05.973355 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.973329    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg" event={"ID":"6200af12-2c64-4113-b61b-e754c746c706","Type":"ContainerStarted","Data":"d9222ef081bb1d0b47081201284fc577c3b45abc159df1db47e844746df60a3e"}
Nov 03 08:15:05.975350 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.975322    5070 scope.go:122] "RemoveContainer" containerID="7b9d84c721107532f48e0d0e796aa280d6c87b21747fe641d297dbad00085568"
Nov 03 08:15:05.975584 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:05.975566    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg_odh-ai-gateway-infra(8a05958e-20e4-4050-a9a8-76c22342d517)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg" podUID="8a05958e-20e4-4050-a9a8-76c22342d517"
Nov 03 08:15:05.976934 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.976911    5070 generic.go:356] "Generic (PLEG): container finished" podID="6923d57c-6c82-4fe8-bd99-15e3be887ce1" containerID="03d12b6bad11846510b86946b63e5647fa3e0064233da4b6371820d9167592e0" exitCode=60
Nov 03 08:15:05.977032 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.976954    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b" event={"ID":"6923d57c-6c82-4fe8-bd99-15e3be887ce1","Type":"ContainerDied","Data":"03d12b6bad11846510b86946b63e5647fa3e0064233da4b6371820d9167592e0"}
Nov 03 08:15:05.977196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.977181    5070 scope.go:122] "RemoveContainer" containerID="03d12b6bad11846510b86946b63e5647fa3e0064233da4b6371820d9167592e0"
Nov 03 08:15:05.977416 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:05.977397    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b_odh-ai-gateway-infra(6923d57c-6c82-4fe8-bd99-15e3be887ce1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b" podUID="6923d57c-6c82-4fe8-bd99-15e3be887ce1"
Nov 03 08:15:05.982074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.982057    5070 scope.go:122] "RemoveContainer" containerID="d25fad067e21fd7a63e2b268f59b9d8d72fc2090fb2cd0f0c515dae26984fc8e"
Nov 03 08:15:05.999030 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:05.999011    5070 scope.go:122] "RemoveContainer" containerID="81b601a0e2b03d773671c1d7a7368b499085d38a20f22daf32b058ea68c40f99"
Nov 03 08:15:06.014220 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:06.014200    5070 scope.go:122] "RemoveContainer" containerID="cedf9ed1e00e369f08570d9c7d811578b59a9d5cbb440b8394a1a3ee509e428a"
Nov 03 08:15:06.030720 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:06.030701    5070 scope.go:122] "RemoveContainer" containerID="66565fb0b62ce496a028f2d6b3aa31fc6c084192a0e7c2a0adef2c46e1278de9"
Nov 03 08:15:06.991141 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:06.991107    5070 generic.go:356] "Generic (PLEG): container finished" podID="754211f3-c239-4469-be61-86b3bf06326c" containerID="53a63b2b0b2df07aa52a6e9e97bb34e213531c7245f6f01f11fa89d1a7216310" exitCode=60
Nov 03 08:15:06.991570 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:06.991149    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk" event={"ID":"754211f3-c239-4469-be61-86b3bf06326c","Type":"ContainerDied","Data":"53a63b2b0b2df07aa52a6e9e97bb34e213531c7245f6f01f11fa89d1a7216310"}
Nov 03 08:15:06.991570 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:06.991262    5070 scope.go:122] "RemoveContainer" containerID="865faf182c801b9554ebf29000ef977068e5878a858f39ed7258826f8e18108f"
Nov 03 08:15:06.991679 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:06.991664    5070 scope.go:122] "RemoveContainer" containerID="53a63b2b0b2df07aa52a6e9e97bb34e213531c7245f6f01f11fa89d1a7216310"
Nov 03 08:15:06.992059 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:06.992041    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk_odh-ai-gateway-infra(754211f3-c239-4469-be61-86b3bf06326c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk" podUID="754211f3-c239-4469-be61-86b3bf06326c"
Nov 03 08:15:06.994394 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:06.994373    5070 scope.go:122] "RemoveContainer" containerID="c890d0b64b747239bf8f59791c408b4e8349527e5eefbac2babf5ebe027e8d6f"
Nov 03 08:15:06.994752 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:06.994732    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm_odh-ai-gateway-infra(e6086e8f-a1b6-43e2-8a92-3a962774083c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm" podUID="e6086e8f-a1b6-43e2-8a92-3a962774083c"
Nov 03 08:15:06.998068 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:06.998040    5070 generic.go:356] "Generic (PLEG): container finished" podID="86e2d036-060e-40b6-8584-bd61b2595169" containerID="ef8cba7a36936b014992ddd694c46a48a9d644f9ea00309ca04a86099a9ba9ac" exitCode=60
Nov 03 08:15:06.998201 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:06.998081    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv" event={"ID":"86e2d036-060e-40b6-8584-bd61b2595169","Type":"ContainerDied","Data":"ef8cba7a36936b014992ddd694c46a48a9d644f9ea00309ca04a86099a9ba9ac"}
Nov 03 08:15:06.998475 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:06.998456    5070 scope.go:122] "RemoveContainer" containerID="ef8cba7a36936b014992ddd694c46a48a9d644f9ea00309ca04a86099a9ba9ac"
Nov 03 08:15:06.998928 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:06.998895    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv_odh-ai-gateway-infra(86e2d036-060e-40b6-8584-bd61b2595169)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv" podUID="86e2d036-060e-40b6-8584-bd61b2595169"
Nov 03 08:15:07.000289 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.000267    5070 generic.go:356] "Generic (PLEG): container finished" podID="6200af12-2c64-4113-b61b-e754c746c706" containerID="d9222ef081bb1d0b47081201284fc577c3b45abc159df1db47e844746df60a3e" exitCode=60
Nov 03 08:15:07.000396 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.000352    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg" event={"ID":"6200af12-2c64-4113-b61b-e754c746c706","Type":"ContainerDied","Data":"d9222ef081bb1d0b47081201284fc577c3b45abc159df1db47e844746df60a3e"}
Nov 03 08:15:07.000603 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.000581    5070 scope.go:122] "RemoveContainer" containerID="d9222ef081bb1d0b47081201284fc577c3b45abc159df1db47e844746df60a3e"
Nov 03 08:15:07.001008 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:07.000981    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg_odh-ai-gateway-infra(6200af12-2c64-4113-b61b-e754c746c706)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg" podUID="6200af12-2c64-4113-b61b-e754c746c706"
Nov 03 08:15:07.002574 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.002540    5070 scope.go:122] "RemoveContainer" containerID="03d12b6bad11846510b86946b63e5647fa3e0064233da4b6371820d9167592e0"
Nov 03 08:15:07.002901 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:07.002875    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b_odh-ai-gateway-infra(6923d57c-6c82-4fe8-bd99-15e3be887ce1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b" podUID="6923d57c-6c82-4fe8-bd99-15e3be887ce1"
Nov 03 08:15:07.004361 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.004345    5070 scope.go:122] "RemoveContainer" containerID="41a84167d752eb6100f2b95c7230b5d58a8f2c7e49108de55dffdba249831688"
Nov 03 08:15:07.004628 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:07.004605    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz_odh-ai-gateway-infra(250bea66-7bc7-41ba-b7a4-2bc0498ed885)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz" podUID="250bea66-7bc7-41ba-b7a4-2bc0498ed885"
Nov 03 08:15:07.006360 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.006337    5070 generic.go:356] "Generic (PLEG): container finished" podID="f228239e-f1d7-4bb8-9fc9-edc630a9df41" containerID="8a4850e9a297800fcefff77ad5193de236fd851949e32e59bb9239ed9885daf9" exitCode=60
Nov 03 08:15:07.006438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.006418    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt" event={"ID":"f228239e-f1d7-4bb8-9fc9-edc630a9df41","Type":"ContainerDied","Data":"8a4850e9a297800fcefff77ad5193de236fd851949e32e59bb9239ed9885daf9"}
Nov 03 08:15:07.006677 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.006643    5070 scope.go:122] "RemoveContainer" containerID="8a4850e9a297800fcefff77ad5193de236fd851949e32e59bb9239ed9885daf9"
Nov 03 08:15:07.006990 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:07.006962    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt_odh-ai-gateway-infra(f228239e-f1d7-4bb8-9fc9-edc630a9df41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt" podUID="f228239e-f1d7-4bb8-9fc9-edc630a9df41"
Nov 03 08:15:07.008602 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.008586    5070 scope.go:122] "RemoveContainer" containerID="9f3eb208f7e3af33df0b83c10f0d4e488f5116159463051ebcb21a0b0dab4c89"
Nov 03 08:15:07.008866 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:07.008848    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894895-rn8xs_odh-ai-gateway-infra(79d22235-c4cd-4701-a8ab-27f8143ce0fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs" podUID="79d22235-c4cd-4701-a8ab-27f8143ce0fa"
Nov 03 08:15:07.009249 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.009229    5070 scope.go:122] "RemoveContainer" containerID="8b72174647f870b6106cbec92778f9b4cf22bc9236a14fe64c2e934ef11e8074"
Nov 03 08:15:07.010659 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.010634    5070 generic.go:356] "Generic (PLEG): container finished" podID="b6cb9d88-cc98-44a7-9e72-ace1b878668f" containerID="c8075c3010e861dde721481f5ffcbfb15dfe3531901610bc9288964cf7eba8e0" exitCode=60
Nov 03 08:15:07.010774 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.010660    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8" event={"ID":"b6cb9d88-cc98-44a7-9e72-ace1b878668f","Type":"ContainerDied","Data":"c8075c3010e861dde721481f5ffcbfb15dfe3531901610bc9288964cf7eba8e0"}
Nov 03 08:15:07.010931 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.010913    5070 scope.go:122] "RemoveContainer" containerID="c8075c3010e861dde721481f5ffcbfb15dfe3531901610bc9288964cf7eba8e0"
Nov 03 08:15:07.011208 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:07.011183    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8_odh-ai-gateway-infra(b6cb9d88-cc98-44a7-9e72-ace1b878668f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8" podUID="b6cb9d88-cc98-44a7-9e72-ace1b878668f"
Nov 03 08:15:07.012846 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.012829    5070 scope.go:122] "RemoveContainer" containerID="dc81580473c252a040840e7b71efcb6388950c83a1c5becdfcdd8810765244bd"
Nov 03 08:15:07.013081 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:07.013058    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd_odh-ai-gateway-infra(6f9a2a18-912c-4366-94b9-96770048ec4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd" podUID="6f9a2a18-912c-4366-94b9-96770048ec4d"
Nov 03 08:15:07.014435 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.014418    5070 scope.go:122] "RemoveContainer" containerID="3d2fe88b7b5bc5ce8220968c3edf3dfd9041f5c8085a02a6aeff987c76afa8f1"
Nov 03 08:15:07.014675 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:07.014654    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f_odh-ai-gateway-infra(8237d5fa-c742-4293-802c-3fc5e29c52f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f" podUID="8237d5fa-c742-4293-802c-3fc5e29c52f0"
Nov 03 08:15:07.016172 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.016149    5070 generic.go:356] "Generic (PLEG): container finished" podID="5a5d91d5-e0e2-41db-bea2-85a814e5274a" containerID="286100bfd9be06845d0d45f5bc2f54f576ab1f3ef10a12d6c0c0c53bcf0a578d" exitCode=60
Nov 03 08:15:07.016258 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.016183    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw" event={"ID":"5a5d91d5-e0e2-41db-bea2-85a814e5274a","Type":"ContainerDied","Data":"286100bfd9be06845d0d45f5bc2f54f576ab1f3ef10a12d6c0c0c53bcf0a578d"}
Nov 03 08:15:07.016459 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.016442    5070 scope.go:122] "RemoveContainer" containerID="286100bfd9be06845d0d45f5bc2f54f576ab1f3ef10a12d6c0c0c53bcf0a578d"
Nov 03 08:15:07.016740 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:07.016719    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw_odh-ai-gateway-infra(5a5d91d5-e0e2-41db-bea2-85a814e5274a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw" podUID="5a5d91d5-e0e2-41db-bea2-85a814e5274a"
Nov 03 08:15:07.035438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.035420    5070 scope.go:122] "RemoveContainer" containerID="9aa2980154ed0252abd894834d5dfab1a859fef3ecf761ba1e35a2945e3ff65e"
Nov 03 08:15:07.059881 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.052771    5070 scope.go:122] "RemoveContainer" containerID="7685c74b17032b94394cac6d52c8c287b829b6cb6bc26599f020f18a37ad0732"
Nov 03 08:15:07.315019 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.295464    5070 scope.go:122] "RemoveContainer" containerID="38e99a89fcd9bb63bd153a961d07b807c86f3d8b5010a1e23e9fc220c5ee0347"
Nov 03 08:15:07.462165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:07.462135    5070 scope.go:122] "RemoveContainer" containerID="b5c6f8522ca7c08b665035ae86ca9d9e0da9c4b47654fc0aa52985fa0e98696c"
Nov 03 08:15:07.714366 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:07.714321    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77803_4112_45fe_bf9b_5ca069a5441f.slice/crio-conmon-71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77803_4112_45fe_bf9b_5ca069a5441f.slice/crio-71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 08:15:08.101977 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:08.101903    5070 scope.go:122] "RemoveContainer" containerID="286100bfd9be06845d0d45f5bc2f54f576ab1f3ef10a12d6c0c0c53bcf0a578d"
Nov 03 08:15:08.103213 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:08.102934    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw_odh-ai-gateway-infra(5a5d91d5-e0e2-41db-bea2-85a814e5274a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw" podUID="5a5d91d5-e0e2-41db-bea2-85a814e5274a"
Nov 03 08:15:08.110232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:08.110211    5070 scope.go:122] "RemoveContainer" containerID="53a63b2b0b2df07aa52a6e9e97bb34e213531c7245f6f01f11fa89d1a7216310"
Nov 03 08:15:08.111678 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:08.111651    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk_odh-ai-gateway-infra(754211f3-c239-4469-be61-86b3bf06326c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk" podUID="754211f3-c239-4469-be61-86b3bf06326c"
Nov 03 08:15:08.112977 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:08.112956    5070 scope.go:122] "RemoveContainer" containerID="ef8cba7a36936b014992ddd694c46a48a9d644f9ea00309ca04a86099a9ba9ac"
Nov 03 08:15:08.113281 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:08.113255    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv_odh-ai-gateway-infra(86e2d036-060e-40b6-8584-bd61b2595169)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv" podUID="86e2d036-060e-40b6-8584-bd61b2595169"
Nov 03 08:15:08.115338 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:08.115273    5070 scope.go:122] "RemoveContainer" containerID="d9222ef081bb1d0b47081201284fc577c3b45abc159df1db47e844746df60a3e"
Nov 03 08:15:08.115575 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:08.115554    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg_odh-ai-gateway-infra(6200af12-2c64-4113-b61b-e754c746c706)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg" podUID="6200af12-2c64-4113-b61b-e754c746c706"
Nov 03 08:15:08.117704 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:08.117683    5070 scope.go:122] "RemoveContainer" containerID="8a4850e9a297800fcefff77ad5193de236fd851949e32e59bb9239ed9885daf9"
Nov 03 08:15:08.118929 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:08.117954    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt_odh-ai-gateway-infra(f228239e-f1d7-4bb8-9fc9-edc630a9df41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt" podUID="f228239e-f1d7-4bb8-9fc9-edc630a9df41"
Nov 03 08:15:08.120122 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:08.120101    5070 scope.go:122] "RemoveContainer" containerID="c8075c3010e861dde721481f5ffcbfb15dfe3531901610bc9288964cf7eba8e0"
Nov 03 08:15:08.120332 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:08.120314    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8_odh-ai-gateway-infra(b6cb9d88-cc98-44a7-9e72-ace1b878668f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8" podUID="b6cb9d88-cc98-44a7-9e72-ace1b878668f"
Nov 03 08:15:10.084950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:10.084915    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5"]
Nov 03 08:15:10.095621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:10.095482    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0531169-lx7z5"]
Nov 03 08:15:12.028532 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:12.028499    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78" path="/var/lib/kubelet/pods/8a02fc3a-0a1a-45a1-a80d-e7afbd0e8b78/volumes"
Nov 03 08:15:13.180653 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:13.180618    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77803_4112_45fe_bf9b_5ca069a5441f.slice/crio-71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a77803_4112_45fe_bf9b_5ca069a5441f.slice/crio-conmon-71953c9e4e43cdd67b1bc07b39bf434ce10046aafc9f9291ee376222fbf5e997.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 08:15:16.018220 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:16.018184    5070 scope.go:122] "RemoveContainer" containerID="7b9d84c721107532f48e0d0e796aa280d6c87b21747fe641d297dbad00085568"
Nov 03 08:15:16.018751 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:16.018269    5070 scope.go:122] "RemoveContainer" containerID="03d12b6bad11846510b86946b63e5647fa3e0064233da4b6371820d9167592e0"
Nov 03 08:15:16.018751 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:16.018388    5070 scope.go:122] "RemoveContainer" containerID="41a84167d752eb6100f2b95c7230b5d58a8f2c7e49108de55dffdba249831688"
Nov 03 08:15:16.798750 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:16.798708    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"]
Nov 03 08:15:16.799566 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:16.799216    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8" podUID="5bb58eee-a968-448c-9204-795ea3d444b7" containerName="istio-proxy" containerID="cri-o://4b209588d42ea1319663a41063f7159c3050fa4cc39e16c967f02958ac542d11" gracePeriod=30
Nov 03 08:15:16.902840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:16.902807    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"]
Nov 03 08:15:16.903042 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:16.903010    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2" podUID="12307292-4369-4f20-9288-9b2005609416" containerName="istio-proxy" containerID="cri-o://8ebc9ee133798b1c6f28bb65926a3dad9df2da4754aebb4c84588ededc84e2c3" gracePeriod=30
Nov 03 08:15:16.908908 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:16.908872    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"]
Nov 03 08:15:16.909128 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:16.909098    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr" podUID="64c976ca-8c6d-4835-b5ca-ba88be627ea7" containerName="istio-proxy" containerID="cri-o://858436e67cd13dbb219a36a6c0c6514844e5c2e2eda96b78203c916c230fc665" gracePeriod=30
Nov 03 08:15:17.018199 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:17.018166    5070 scope.go:122] "RemoveContainer" containerID="dc81580473c252a040840e7b71efcb6388950c83a1c5becdfcdd8810765244bd"
Nov 03 08:15:17.018453 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:17.018364    5070 scope.go:122] "RemoveContainer" containerID="ef8cba7a36936b014992ddd694c46a48a9d644f9ea00309ca04a86099a9ba9ac"
Nov 03 08:15:17.019198 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:17.019131    5070 scope.go:122] "RemoveContainer" containerID="9f3eb208f7e3af33df0b83c10f0d4e488f5116159463051ebcb21a0b0dab4c89"
Nov 03 08:15:17.436788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:17.434239    5070 generic.go:356] "Generic (PLEG): container finished" podID="8a05958e-20e4-4050-a9a8-76c22342d517" containerID="e59115f1b5687f9e04d7843dd519338adecb3f5ea1c153a6efc03523b8907718" exitCode=60
Nov 03 08:15:17.436788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:17.434431    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg" event={"ID":"8a05958e-20e4-4050-a9a8-76c22342d517","Type":"ContainerDied","Data":"e59115f1b5687f9e04d7843dd519338adecb3f5ea1c153a6efc03523b8907718"}
Nov 03 08:15:17.436788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:17.434513    5070 scope.go:122] "RemoveContainer" containerID="7b9d84c721107532f48e0d0e796aa280d6c87b21747fe641d297dbad00085568"
Nov 03 08:15:17.453780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:17.451411    5070 scope.go:122] "RemoveContainer" containerID="e59115f1b5687f9e04d7843dd519338adecb3f5ea1c153a6efc03523b8907718"
Nov 03 08:15:17.453780 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:17.452463    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg_odh-ai-gateway-infra(8a05958e-20e4-4050-a9a8-76c22342d517)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg" podUID="8a05958e-20e4-4050-a9a8-76c22342d517"
Nov 03 08:15:17.476587 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:17.475014    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b" event={"ID":"6923d57c-6c82-4fe8-bd99-15e3be887ce1","Type":"ContainerStarted","Data":"35e59331f17d60eaa8ba8e247a354a4558565b1e2d9a8177b58f3badfc0c7a4a"}
Nov 03 08:15:17.476587 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:17.475518    5070 scope.go:122] "RemoveContainer" containerID="35e59331f17d60eaa8ba8e247a354a4558565b1e2d9a8177b58f3badfc0c7a4a"
Nov 03 08:15:17.476587 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:17.475987    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b_odh-ai-gateway-infra(6923d57c-6c82-4fe8-bd99-15e3be887ce1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b" podUID="6923d57c-6c82-4fe8-bd99-15e3be887ce1"
Nov 03 08:15:17.524322 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:17.513841    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz" event={"ID":"250bea66-7bc7-41ba-b7a4-2bc0498ed885","Type":"ContainerStarted","Data":"6ebf88ad5b3356606f38f63491f1724367a2126129b7cc3598c1e975b12d8923"}
Nov 03 08:15:17.524322 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:17.514288    5070 scope.go:122] "RemoveContainer" containerID="6ebf88ad5b3356606f38f63491f1724367a2126129b7cc3598c1e975b12d8923"
Nov 03 08:15:17.524322 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:17.514671    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz_odh-ai-gateway-infra(250bea66-7bc7-41ba-b7a4-2bc0498ed885)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz" podUID="250bea66-7bc7-41ba-b7a4-2bc0498ed885"
Nov 03 08:15:18.032733 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.032694    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"]
Nov 03 08:15:18.034872 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.034753    5070 scope.go:122] "RemoveContainer" containerID="3d2fe88b7b5bc5ce8220968c3edf3dfd9041f5c8085a02a6aeff987c76afa8f1"
Nov 03 08:15:18.039060 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.039039    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.047539 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.047522    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-99410b-openshift-default-dockercfg-64wgq\""
Nov 03 08:15:18.056437 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.056413    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"]
Nov 03 08:15:18.144750 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.144716    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wht9m\" (UniqueName: \"kubernetes.io/projected/ae54e276-2450-4cef-98d9-e6cc85ed028a-kube-api-access-wht9m\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.145004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.144964    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-workload-certs\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.145067 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.145054    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-workload-socket\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.145130 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.145116    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-token\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.145196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.145179    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae54e276-2450-4cef-98d9-e6cc85ed028a-istiod-ca-cert\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.145312 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.145283    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-data\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.145391 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.145374    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-podinfo\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.145467 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.145455    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-credential-socket\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.145518 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.145481    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-envoy\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.249960 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.249926    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-credential-socket\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.250164 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.249970    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-envoy\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.250164 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.250031    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wht9m\" (UniqueName: \"kubernetes.io/projected/ae54e276-2450-4cef-98d9-e6cc85ed028a-kube-api-access-wht9m\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.250303 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.250175    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-workload-certs\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.250303 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.250249    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-workload-socket\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.250303 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.250286    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-token\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.250484 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.250343    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae54e276-2450-4cef-98d9-e6cc85ed028a-istiod-ca-cert\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.250484 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.250368    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-data\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.250484 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.250420    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-podinfo\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.251607 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.251223    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-workload-certs\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.251607 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.251544    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-workload-socket\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.251902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.251773    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-data\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.251902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.251875    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-credential-socket\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.252210 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.252169    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae54e276-2450-4cef-98d9-e6cc85ed028a-istiod-ca-cert\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.262018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.261991    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-envoy\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.262355 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.262339    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-podinfo\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.262951 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.262930    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wht9m\" (UniqueName: \"kubernetes.io/projected/ae54e276-2450-4cef-98d9-e6cc85ed028a-kube-api-access-wht9m\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.263497 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.263473    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-token\") pod \"e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") " pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.381267 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.381225    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:18.443788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.443737    5070 patch_prober.go:29] interesting pod/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:15:18.443962 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.443814    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2" podUID="12307292-4369-4f20-9288-9b2005609416" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:15:18.483547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.483430    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg"]
Nov 03 08:15:18.536728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.536445    5070 generic.go:356] "Generic (PLEG): container finished" podID="6f9a2a18-912c-4366-94b9-96770048ec4d" containerID="26db5d7682e0f7de4174b4fff0d3daf5ad0c01c3acb5275f6b741ec4daa389e3" exitCode=60
Nov 03 08:15:18.536728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.536628    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd" event={"ID":"6f9a2a18-912c-4366-94b9-96770048ec4d","Type":"ContainerDied","Data":"26db5d7682e0f7de4174b4fff0d3daf5ad0c01c3acb5275f6b741ec4daa389e3"}
Nov 03 08:15:18.536728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.536691    5070 scope.go:122] "RemoveContainer" containerID="dc81580473c252a040840e7b71efcb6388950c83a1c5becdfcdd8810765244bd"
Nov 03 08:15:18.539126 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.537264    5070 scope.go:122] "RemoveContainer" containerID="26db5d7682e0f7de4174b4fff0d3daf5ad0c01c3acb5275f6b741ec4daa389e3"
Nov 03 08:15:18.539126 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:18.538038    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd_odh-ai-gateway-infra(6f9a2a18-912c-4366-94b9-96770048ec4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd" podUID="6f9a2a18-912c-4366-94b9-96770048ec4d"
Nov 03 08:15:18.545611 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.545554    5070 generic.go:356] "Generic (PLEG): container finished" podID="86e2d036-060e-40b6-8584-bd61b2595169" containerID="ab43522f36bce78677b03e574ca71128fd0258adeb98d2364099a55e2a1e8e73" exitCode=60
Nov 03 08:15:18.545894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.545870    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv" event={"ID":"86e2d036-060e-40b6-8584-bd61b2595169","Type":"ContainerDied","Data":"ab43522f36bce78677b03e574ca71128fd0258adeb98d2364099a55e2a1e8e73"}
Nov 03 08:15:18.546504 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.546484    5070 scope.go:122] "RemoveContainer" containerID="ab43522f36bce78677b03e574ca71128fd0258adeb98d2364099a55e2a1e8e73"
Nov 03 08:15:18.547448 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:18.547421    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv_odh-ai-gateway-infra(86e2d036-060e-40b6-8584-bd61b2595169)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv" podUID="86e2d036-060e-40b6-8584-bd61b2595169"
Nov 03 08:15:18.556782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.555903    5070 generic.go:356] "Generic (PLEG): container finished" podID="6923d57c-6c82-4fe8-bd99-15e3be887ce1" containerID="35e59331f17d60eaa8ba8e247a354a4558565b1e2d9a8177b58f3badfc0c7a4a" exitCode=60
Nov 03 08:15:18.556782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.555991    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b" event={"ID":"6923d57c-6c82-4fe8-bd99-15e3be887ce1","Type":"ContainerDied","Data":"35e59331f17d60eaa8ba8e247a354a4558565b1e2d9a8177b58f3badfc0c7a4a"}
Nov 03 08:15:18.556782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.556440    5070 scope.go:122] "RemoveContainer" containerID="35e59331f17d60eaa8ba8e247a354a4558565b1e2d9a8177b58f3badfc0c7a4a"
Nov 03 08:15:18.557707 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:18.557678    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b_odh-ai-gateway-infra(6923d57c-6c82-4fe8-bd99-15e3be887ce1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b" podUID="6923d57c-6c82-4fe8-bd99-15e3be887ce1"
Nov 03 08:15:18.558696 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.558546    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b"]
Nov 03 08:15:18.565970 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.565950    5070 scope.go:122] "RemoveContainer" containerID="ef8cba7a36936b014992ddd694c46a48a9d644f9ea00309ca04a86099a9ba9ac"
Nov 03 08:15:18.567282 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.567256    5070 generic.go:356] "Generic (PLEG): container finished" podID="250bea66-7bc7-41ba-b7a4-2bc0498ed885" containerID="6ebf88ad5b3356606f38f63491f1724367a2126129b7cc3598c1e975b12d8923" exitCode=60
Nov 03 08:15:18.567956 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.567293    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz" event={"ID":"250bea66-7bc7-41ba-b7a4-2bc0498ed885","Type":"ContainerDied","Data":"6ebf88ad5b3356606f38f63491f1724367a2126129b7cc3598c1e975b12d8923"}
Nov 03 08:15:18.569067 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.568206    5070 scope.go:122] "RemoveContainer" containerID="6ebf88ad5b3356606f38f63491f1724367a2126129b7cc3598c1e975b12d8923"
Nov 03 08:15:18.569067 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:18.568635    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz_odh-ai-gateway-infra(250bea66-7bc7-41ba-b7a4-2bc0498ed885)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz" podUID="250bea66-7bc7-41ba-b7a4-2bc0498ed885"
Nov 03 08:15:18.572598 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.571623    5070 generic.go:356] "Generic (PLEG): container finished" podID="79d22235-c4cd-4701-a8ab-27f8143ce0fa" containerID="d22c8aa23d7204a40e48f7974d6b4334faf46f583232ad9a544bf7cf740a9a15" exitCode=60
Nov 03 08:15:18.572598 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.571727    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs" event={"ID":"79d22235-c4cd-4701-a8ab-27f8143ce0fa","Type":"ContainerDied","Data":"d22c8aa23d7204a40e48f7974d6b4334faf46f583232ad9a544bf7cf740a9a15"}
Nov 03 08:15:18.572598 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.572188    5070 scope.go:122] "RemoveContainer" containerID="d22c8aa23d7204a40e48f7974d6b4334faf46f583232ad9a544bf7cf740a9a15"
Nov 03 08:15:18.572598 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:18.572552    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894895-rn8xs_odh-ai-gateway-infra(79d22235-c4cd-4701-a8ab-27f8143ce0fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs" podUID="79d22235-c4cd-4701-a8ab-27f8143ce0fa"
Nov 03 08:15:18.585124 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.585104    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz"]
Nov 03 08:15:18.616694 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.616663    5070 scope.go:122] "RemoveContainer" containerID="03d12b6bad11846510b86946b63e5647fa3e0064233da4b6371820d9167592e0"
Nov 03 08:15:18.641601 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.641580    5070 scope.go:122] "RemoveContainer" containerID="41a84167d752eb6100f2b95c7230b5d58a8f2c7e49108de55dffdba249831688"
Nov 03 08:15:18.674233 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.674218    5070 scope.go:122] "RemoveContainer" containerID="9f3eb208f7e3af33df0b83c10f0d4e488f5116159463051ebcb21a0b0dab4c89"
Nov 03 08:15:18.755257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.755226    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"]
Nov 03 08:15:18.756151 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:15:18.756125    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae54e276_2450_4cef_98d9_e6cc85ed028a.slice/crio-e049643897188cf5e03d0618d7dd9d68476592b375aea666b2d1ab505f19a17d WatchSource:0}: Error finding container e049643897188cf5e03d0618d7dd9d68476592b375aea666b2d1ab505f19a17d: Status 404 returned error can't find the container with id e049643897188cf5e03d0618d7dd9d68476592b375aea666b2d1ab505f19a17d
Nov 03 08:15:18.759326 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.759290    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:15:18.759456 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.759384    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:15:18.759456 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.759431    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:15:18.800751 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.800704    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg"
Nov 03 08:15:18.807833 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.807804    5070 patch_prober.go:29] interesting pod/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:15:18.807950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.807869    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8" podUID="5bb58eee-a968-448c-9204-795ea3d444b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:15:18.988909 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.985733    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a05958e-20e4-4050-a9a8-76c22342d517-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a05958e-20e4-4050-a9a8-76c22342d517-maas-api-service-ca\") pod \"8a05958e-20e4-4050-a9a8-76c22342d517\" (UID: \"8a05958e-20e4-4050-a9a8-76c22342d517\") "
Nov 03 08:15:18.989292 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:18.989261    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a05958e-20e4-4050-a9a8-76c22342d517-kube-api-access-pt2rk\" (UniqueName: \"kubernetes.io/projected/8a05958e-20e4-4050-a9a8-76c22342d517-kube-api-access-pt2rk\") pod \"8a05958e-20e4-4050-a9a8-76c22342d517\" (UID: \"8a05958e-20e4-4050-a9a8-76c22342d517\") "
Nov 03 08:15:19.002288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:19.002249    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a05958e-20e4-4050-a9a8-76c22342d517-maas-api-service-ca" pod "8a05958e-20e4-4050-a9a8-76c22342d517" (UID: "8a05958e-20e4-4050-a9a8-76c22342d517"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:19.018246 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:19.018215    5070 scope.go:122] "RemoveContainer" containerID="c8075c3010e861dde721481f5ffcbfb15dfe3531901610bc9288964cf7eba8e0"
Nov 03 08:15:19.019673 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:19.019647    5070 scope.go:122] "RemoveContainer" containerID="8a4850e9a297800fcefff77ad5193de236fd851949e32e59bb9239ed9885daf9"
Nov 03 08:15:19.022875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:19.022847    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a05958e-20e4-4050-a9a8-76c22342d517-kube-api-access-pt2rk" pod "8a05958e-20e4-4050-a9a8-76c22342d517" (UID: "8a05958e-20e4-4050-a9a8-76c22342d517"). InnerVolumeSpecName "kube-api-access-pt2rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:19.111065 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:19.111031    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a05958e-20e4-4050-a9a8-76c22342d517-maas-api-service-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:19.111065 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:19.111060    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt2rk\" (UniqueName: \"kubernetes.io/projected/8a05958e-20e4-4050-a9a8-76c22342d517-kube-api-access-pt2rk\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:19.570971 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:19.570945    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd"]
Nov 03 08:15:19.589901 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:19.589870    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt" event={"ID":"f228239e-f1d7-4bb8-9fc9-edc630a9df41","Type":"ContainerStarted","Data":"35a493ed5a09c6decda068878efe220c3f245a4aa3316c1b9b661a4901923be2"}
Nov 03 08:15:19.599606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:19.599564    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k" event={"ID":"ae54e276-2450-4cef-98d9-e6cc85ed028a","Type":"ContainerStarted","Data":"b352c8ba7f3f94e3612f9c06f03b6efbd0508e20d4b388ee2bd4fddbfce28315"}
Nov 03 08:15:19.599887 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:19.599657    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k" event={"ID":"ae54e276-2450-4cef-98d9-e6cc85ed028a","Type":"ContainerStarted","Data":"e049643897188cf5e03d0618d7dd9d68476592b375aea666b2d1ab505f19a17d"}
Nov 03 08:15:19.619370 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:19.619345    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv"]
Nov 03 08:15:19.625670 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:19.625638    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8" event={"ID":"b6cb9d88-cc98-44a7-9e72-ace1b878668f","Type":"ContainerStarted","Data":"b1a84e71c4a6cb9a752aa513b5341e546fb2551598d8acfb5bf51855915868d1"}
Nov 03 08:15:19.644771 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:19.641089    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k" podStartSLOduration=2.641071884 podStartE2EDuration="2.641071884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:19.635989299 +0000 UTC m=+1317.750453234" watchObservedRunningTime="2026-11-03 08:15:19.641071884 +0000 UTC m=+1317.755535817"
Nov 03 08:15:19.644771 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:19.642105    5070 generic.go:356] "Generic (PLEG): container finished" podID="8237d5fa-c742-4293-802c-3fc5e29c52f0" containerID="d79ce00f79bc52d3fcc43acd690ba21cce18ffd546f268a832d87a13f830b686" exitCode=60
Nov 03 08:15:19.644771 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:19.642258    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f" event={"ID":"8237d5fa-c742-4293-802c-3fc5e29c52f0","Type":"ContainerDied","Data":"d79ce00f79bc52d3fcc43acd690ba21cce18ffd546f268a832d87a13f830b686"}
Nov 03 08:15:19.644771 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:19.642302    5070 scope.go:122] "RemoveContainer" containerID="3d2fe88b7b5bc5ce8220968c3edf3dfd9041f5c8085a02a6aeff987c76afa8f1"
Nov 03 08:15:19.644771 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:19.644520    5070 scope.go:122] "RemoveContainer" containerID="d79ce00f79bc52d3fcc43acd690ba21cce18ffd546f268a832d87a13f830b686"
Nov 03 08:15:19.645312 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:19.645286    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f_odh-ai-gateway-infra(8237d5fa-c742-4293-802c-3fc5e29c52f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f" podUID="8237d5fa-c742-4293-802c-3fc5e29c52f0"
Nov 03 08:15:19.659605 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:19.659585    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg" event={"ID":"8a05958e-20e4-4050-a9a8-76c22342d517","Type":"ContainerDied","Data":"782cf33f3a9b1bbcb491d06f655a62ebc9e6befc2848c544e8c9cfae20a49672"}
Nov 03 08:15:19.659804 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:19.659781    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg"
Nov 03 08:15:19.669636 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:19.669604    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs"]
Nov 03 08:15:19.695402 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:19.695300    5070 scope.go:122] "RemoveContainer" containerID="e59115f1b5687f9e04d7843dd519338adecb3f5ea1c153a6efc03523b8907718"
Nov 03 08:15:20.114002 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.113973    5070 scope.go:122] "RemoveContainer" containerID="d9222ef081bb1d0b47081201284fc577c3b45abc159df1db47e844746df60a3e"
Nov 03 08:15:20.114997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.114974    5070 scope.go:122] "RemoveContainer" containerID="286100bfd9be06845d0d45f5bc2f54f576ab1f3ef10a12d6c0c0c53bcf0a578d"
Nov 03 08:15:20.115275 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.115256    5070 scope.go:122] "RemoveContainer" containerID="c890d0b64b747239bf8f59791c408b4e8349527e5eefbac2babf5ebe027e8d6f"
Nov 03 08:15:20.128289 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.128266    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg"]
Nov 03 08:15:20.145923 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.144260    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-3e88b7-29894895-hhwcg"]
Nov 03 08:15:20.335633 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.335606    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd"
Nov 03 08:15:20.417151 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.417011    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f9a2a18-912c-4366-94b9-96770048ec4d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9a2a18-912c-4366-94b9-96770048ec4d-maas-api-service-ca\") pod \"6f9a2a18-912c-4366-94b9-96770048ec4d\" (UID: \"6f9a2a18-912c-4366-94b9-96770048ec4d\") "
Nov 03 08:15:20.417542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.417415    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f9a2a18-912c-4366-94b9-96770048ec4d-kube-api-access-vwsl9\" (UniqueName: \"kubernetes.io/projected/6f9a2a18-912c-4366-94b9-96770048ec4d-kube-api-access-vwsl9\") pod \"6f9a2a18-912c-4366-94b9-96770048ec4d\" (UID: \"6f9a2a18-912c-4366-94b9-96770048ec4d\") "
Nov 03 08:15:20.490435 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.490386    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f9a2a18-912c-4366-94b9-96770048ec4d-maas-api-service-ca" pod "6f9a2a18-912c-4366-94b9-96770048ec4d" (UID: "6f9a2a18-912c-4366-94b9-96770048ec4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:20.493965 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.493934    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f9a2a18-912c-4366-94b9-96770048ec4d-maas-api-service-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:20.495830 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.495800    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:20.512184 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.511261    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:20.521387 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.521358    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs"
Nov 03 08:15:20.523693 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.523620    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv"
Nov 03 08:15:20.563100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.563011    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9a2a18-912c-4366-94b9-96770048ec4d-kube-api-access-vwsl9" pod "6f9a2a18-912c-4366-94b9-96770048ec4d" (UID: "6f9a2a18-912c-4366-94b9-96770048ec4d"). InnerVolumeSpecName "kube-api-access-vwsl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:20.597039 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.596479    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwsl9\" (UniqueName: \"kubernetes.io/projected/6f9a2a18-912c-4366-94b9-96770048ec4d-kube-api-access-vwsl9\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:20.612553 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.612524    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b"
Nov 03 08:15:20.619049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.619019    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz"
Nov 03 08:15:20.626971 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.626941    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt"]
Nov 03 08:15:20.698637 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.697109    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d22235-c4cd-4701-a8ab-27f8143ce0fa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79d22235-c4cd-4701-a8ab-27f8143ce0fa-maas-api-service-ca\") pod \"79d22235-c4cd-4701-a8ab-27f8143ce0fa\" (UID: \"79d22235-c4cd-4701-a8ab-27f8143ce0fa\") "
Nov 03 08:15:20.698637 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.697457    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86e2d036-060e-40b6-8584-bd61b2595169-kube-api-access-89nsf\" (UniqueName: \"kubernetes.io/projected/86e2d036-060e-40b6-8584-bd61b2595169-kube-api-access-89nsf\") pod \"86e2d036-060e-40b6-8584-bd61b2595169\" (UID: \"86e2d036-060e-40b6-8584-bd61b2595169\") "
Nov 03 08:15:20.698637 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.697515    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d22235-c4cd-4701-a8ab-27f8143ce0fa-kube-api-access-4b9bn\" (UniqueName: \"kubernetes.io/projected/79d22235-c4cd-4701-a8ab-27f8143ce0fa-kube-api-access-4b9bn\") pod \"79d22235-c4cd-4701-a8ab-27f8143ce0fa\" (UID: \"79d22235-c4cd-4701-a8ab-27f8143ce0fa\") "
Nov 03 08:15:20.698637 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.697566    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86e2d036-060e-40b6-8584-bd61b2595169-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86e2d036-060e-40b6-8584-bd61b2595169-maas-api-service-ca\") pod \"86e2d036-060e-40b6-8584-bd61b2595169\" (UID: \"86e2d036-060e-40b6-8584-bd61b2595169\") "
Nov 03 08:15:20.699404 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.699163    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86e2d036-060e-40b6-8584-bd61b2595169-maas-api-service-ca" pod "86e2d036-060e-40b6-8584-bd61b2595169" (UID: "86e2d036-060e-40b6-8584-bd61b2595169"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:20.699962 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.699652    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d22235-c4cd-4701-a8ab-27f8143ce0fa-maas-api-service-ca" pod "79d22235-c4cd-4701-a8ab-27f8143ce0fa" (UID: "79d22235-c4cd-4701-a8ab-27f8143ce0fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:20.709435 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.709202    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv" event={"ID":"86e2d036-060e-40b6-8584-bd61b2595169","Type":"ContainerDied","Data":"e0cdc3ef9810725602a09af8d6e65d029be36a6080668c3e9ac7bdab408b68e0"}
Nov 03 08:15:20.709435 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.709270    5070 scope.go:122] "RemoveContainer" containerID="ab43522f36bce78677b03e574ca71128fd0258adeb98d2364099a55e2a1e8e73"
Nov 03 08:15:20.709435 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.709413    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv"
Nov 03 08:15:20.709828 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.709805    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d22235-c4cd-4701-a8ab-27f8143ce0fa-kube-api-access-4b9bn" pod "79d22235-c4cd-4701-a8ab-27f8143ce0fa" (UID: "79d22235-c4cd-4701-a8ab-27f8143ce0fa"). InnerVolumeSpecName "kube-api-access-4b9bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:20.713786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.713724    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e2d036-060e-40b6-8584-bd61b2595169-kube-api-access-89nsf" pod "86e2d036-060e-40b6-8584-bd61b2595169" (UID: "86e2d036-060e-40b6-8584-bd61b2595169"). InnerVolumeSpecName "kube-api-access-89nsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:20.715457 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.715398    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b" event={"ID":"6923d57c-6c82-4fe8-bd99-15e3be887ce1","Type":"ContainerDied","Data":"334628b19ffe23d434a54f0791f9c92a7639f40463527baab9dd6259906ec895"}
Nov 03 08:15:20.717345 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.716483    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b"
Nov 03 08:15:20.722066 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.719842    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz" event={"ID":"250bea66-7bc7-41ba-b7a4-2bc0498ed885","Type":"ContainerDied","Data":"e18eed1fc752b4b0fc8e50b15fa9710101e7ff19224789207583386a877f3de5"}
Nov 03 08:15:20.722066 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.719930    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz"
Nov 03 08:15:20.727991 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.727669    5070 generic.go:356] "Generic (PLEG): container finished" podID="f228239e-f1d7-4bb8-9fc9-edc630a9df41" containerID="35a493ed5a09c6decda068878efe220c3f245a4aa3316c1b9b661a4901923be2" exitCode=60
Nov 03 08:15:20.727991 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.727817    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt" event={"ID":"f228239e-f1d7-4bb8-9fc9-edc630a9df41","Type":"ContainerDied","Data":"35a493ed5a09c6decda068878efe220c3f245a4aa3316c1b9b661a4901923be2"}
Nov 03 08:15:20.755408 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.755367    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8"]
Nov 03 08:15:20.771540 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.771507    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs" event={"ID":"79d22235-c4cd-4701-a8ab-27f8143ce0fa","Type":"ContainerDied","Data":"5d0f80cd393dc1e0d7659d9b2df7b18ca5e71a39a8ab6d5590957a0a496db5af"}
Nov 03 08:15:20.771768 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.771631    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs"
Nov 03 08:15:20.775552 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.775528    5070 scope.go:122] "RemoveContainer" containerID="35e59331f17d60eaa8ba8e247a354a4558565b1e2d9a8177b58f3badfc0c7a4a"
Nov 03 08:15:20.786949 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.786922    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f"]
Nov 03 08:15:20.802167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.800514    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6923d57c-6c82-4fe8-bd99-15e3be887ce1-kube-api-access-sgqk4\" (UniqueName: \"kubernetes.io/projected/6923d57c-6c82-4fe8-bd99-15e3be887ce1-kube-api-access-sgqk4\") pod \"6923d57c-6c82-4fe8-bd99-15e3be887ce1\" (UID: \"6923d57c-6c82-4fe8-bd99-15e3be887ce1\") "
Nov 03 08:15:20.802167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.800611    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/250bea66-7bc7-41ba-b7a4-2bc0498ed885-kube-api-access-w6zhk\" (UniqueName: \"kubernetes.io/projected/250bea66-7bc7-41ba-b7a4-2bc0498ed885-kube-api-access-w6zhk\") pod \"250bea66-7bc7-41ba-b7a4-2bc0498ed885\" (UID: \"250bea66-7bc7-41ba-b7a4-2bc0498ed885\") "
Nov 03 08:15:20.802167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.800701    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/250bea66-7bc7-41ba-b7a4-2bc0498ed885-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/250bea66-7bc7-41ba-b7a4-2bc0498ed885-maas-api-service-ca\") pod \"250bea66-7bc7-41ba-b7a4-2bc0498ed885\" (UID: \"250bea66-7bc7-41ba-b7a4-2bc0498ed885\") "
Nov 03 08:15:20.802167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.801053    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6923d57c-6c82-4fe8-bd99-15e3be887ce1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6923d57c-6c82-4fe8-bd99-15e3be887ce1-maas-api-service-ca\") pod \"6923d57c-6c82-4fe8-bd99-15e3be887ce1\" (UID: \"6923d57c-6c82-4fe8-bd99-15e3be887ce1\") "
Nov 03 08:15:20.803062 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.802816    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89nsf\" (UniqueName: \"kubernetes.io/projected/86e2d036-060e-40b6-8584-bd61b2595169-kube-api-access-89nsf\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:20.803062 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.802845    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b9bn\" (UniqueName: \"kubernetes.io/projected/79d22235-c4cd-4701-a8ab-27f8143ce0fa-kube-api-access-4b9bn\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:20.803062 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.802859    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86e2d036-060e-40b6-8584-bd61b2595169-maas-api-service-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:20.803062 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.802872    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79d22235-c4cd-4701-a8ab-27f8143ce0fa-maas-api-service-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:20.803672 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.803443    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6923d57c-6c82-4fe8-bd99-15e3be887ce1-maas-api-service-ca" pod "6923d57c-6c82-4fe8-bd99-15e3be887ce1" (UID: "6923d57c-6c82-4fe8-bd99-15e3be887ce1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:20.804040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.803922    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250bea66-7bc7-41ba-b7a4-2bc0498ed885-maas-api-service-ca" pod "250bea66-7bc7-41ba-b7a4-2bc0498ed885" (UID: "250bea66-7bc7-41ba-b7a4-2bc0498ed885"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:20.805744 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.805352    5070 scope.go:122] "RemoveContainer" containerID="6ebf88ad5b3356606f38f63491f1724367a2126129b7cc3598c1e975b12d8923"
Nov 03 08:15:20.805744 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.805529    5070 generic.go:356] "Generic (PLEG): container finished" podID="b6cb9d88-cc98-44a7-9e72-ace1b878668f" containerID="b1a84e71c4a6cb9a752aa513b5341e546fb2551598d8acfb5bf51855915868d1" exitCode=60
Nov 03 08:15:20.805744 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.805697    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8" event={"ID":"b6cb9d88-cc98-44a7-9e72-ace1b878668f","Type":"ContainerDied","Data":"b1a84e71c4a6cb9a752aa513b5341e546fb2551598d8acfb5bf51855915868d1"}
Nov 03 08:15:20.808561 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.808437    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250bea66-7bc7-41ba-b7a4-2bc0498ed885-kube-api-access-w6zhk" pod "250bea66-7bc7-41ba-b7a4-2bc0498ed885" (UID: "250bea66-7bc7-41ba-b7a4-2bc0498ed885"). InnerVolumeSpecName "kube-api-access-w6zhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:20.812092 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.812069    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd" event={"ID":"6f9a2a18-912c-4366-94b9-96770048ec4d","Type":"ContainerDied","Data":"87be978c8ecacad2e19d064008fe95909037a41f6395a87d23227d75efe88ddd"}
Nov 03 08:15:20.814040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.812304    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd"
Nov 03 08:15:20.818888 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.818864    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:20.819932 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.819916    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:15:20.820467 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.820438    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6923d57c-6c82-4fe8-bd99-15e3be887ce1-kube-api-access-sgqk4" pod "6923d57c-6c82-4fe8-bd99-15e3be887ce1" (UID: "6923d57c-6c82-4fe8-bd99-15e3be887ce1"). InnerVolumeSpecName "kube-api-access-sgqk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:20.830261 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.830235    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs"]
Nov 03 08:15:20.841800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.841778    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894895-rn8xs"]
Nov 03 08:15:20.851627 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.851474    5070 scope.go:122] "RemoveContainer" containerID="8a4850e9a297800fcefff77ad5193de236fd851949e32e59bb9239ed9885daf9"
Nov 03 08:15:20.905732 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.905626    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6923d57c-6c82-4fe8-bd99-15e3be887ce1-maas-api-service-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:20.905732 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.905661    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgqk4\" (UniqueName: \"kubernetes.io/projected/6923d57c-6c82-4fe8-bd99-15e3be887ce1-kube-api-access-sgqk4\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:20.905732 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.905679    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6zhk\" (UniqueName: \"kubernetes.io/projected/250bea66-7bc7-41ba-b7a4-2bc0498ed885-kube-api-access-w6zhk\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:20.905732 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.905694    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/250bea66-7bc7-41ba-b7a4-2bc0498ed885-maas-api-service-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:20.949528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:20.948138    5070 scope.go:122] "RemoveContainer" containerID="d22c8aa23d7204a40e48f7974d6b4334faf46f583232ad9a544bf7cf740a9a15"
Nov 03 08:15:21.011314 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.011276    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd"]
Nov 03 08:15:21.032946 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.032908    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-fe584a-29894895-2bgbd"]
Nov 03 08:15:21.060597 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.060566    5070 scope.go:122] "RemoveContainer" containerID="53a63b2b0b2df07aa52a6e9e97bb34e213531c7245f6f01f11fa89d1a7216310"
Nov 03 08:15:21.074375 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.074351    5070 scope.go:122] "RemoveContainer" containerID="c8075c3010e861dde721481f5ffcbfb15dfe3531901610bc9288964cf7eba8e0"
Nov 03 08:15:21.089106 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.089077    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt"
Nov 03 08:15:21.120127 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.120086    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b"]
Nov 03 08:15:21.164378 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.164348    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ed5d1a-29894895-4wb2b"]
Nov 03 08:15:21.165905 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.165878    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz"]
Nov 03 08:15:21.167715 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.167689    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f228239e-f1d7-4bb8-9fc9-edc630a9df41-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f228239e-f1d7-4bb8-9fc9-edc630a9df41-maas-api-service-ca\") pod \"f228239e-f1d7-4bb8-9fc9-edc630a9df41\" (UID: \"f228239e-f1d7-4bb8-9fc9-edc630a9df41\") "
Nov 03 08:15:21.168125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.168095    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f228239e-f1d7-4bb8-9fc9-edc630a9df41-kube-api-access-vmv6j\" (UniqueName: \"kubernetes.io/projected/f228239e-f1d7-4bb8-9fc9-edc630a9df41-kube-api-access-vmv6j\") pod \"f228239e-f1d7-4bb8-9fc9-edc630a9df41\" (UID: \"f228239e-f1d7-4bb8-9fc9-edc630a9df41\") "
Nov 03 08:15:21.170638 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.170615    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f228239e-f1d7-4bb8-9fc9-edc630a9df41-maas-api-service-ca" pod "f228239e-f1d7-4bb8-9fc9-edc630a9df41" (UID: "f228239e-f1d7-4bb8-9fc9-edc630a9df41"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:21.173076 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.173051    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f228239e-f1d7-4bb8-9fc9-edc630a9df41-kube-api-access-vmv6j" pod "f228239e-f1d7-4bb8-9fc9-edc630a9df41" (UID: "f228239e-f1d7-4bb8-9fc9-edc630a9df41"). InnerVolumeSpecName "kube-api-access-vmv6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:21.180208 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.180190    5070 scope.go:122] "RemoveContainer" containerID="26db5d7682e0f7de4174b4fff0d3daf5ad0c01c3acb5275f6b741ec4daa389e3"
Nov 03 08:15:21.212252 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.212194    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-ffe7a8-29894895-4m8kz"]
Nov 03 08:15:21.241828 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.241803    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv"]
Nov 03 08:15:21.248162 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.248132    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-19bb0a-29894895-5m2cv"]
Nov 03 08:15:21.270691 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.270660    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmv6j\" (UniqueName: \"kubernetes.io/projected/f228239e-f1d7-4bb8-9fc9-edc630a9df41-kube-api-access-vmv6j\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:21.270839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.270827    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f228239e-f1d7-4bb8-9fc9-edc630a9df41-maas-api-service-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:21.416228 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.416201    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8"
Nov 03 08:15:21.419073 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.419055    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f"
Nov 03 08:15:21.478513 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.478481    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6cb9d88-cc98-44a7-9e72-ace1b878668f-kube-api-access-b8gzq\" (UniqueName: \"kubernetes.io/projected/b6cb9d88-cc98-44a7-9e72-ace1b878668f-kube-api-access-b8gzq\") pod \"b6cb9d88-cc98-44a7-9e72-ace1b878668f\" (UID: \"b6cb9d88-cc98-44a7-9e72-ace1b878668f\") "
Nov 03 08:15:21.478742 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.478714    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8237d5fa-c742-4293-802c-3fc5e29c52f0-kube-api-access-m6dnj\" (UniqueName: \"kubernetes.io/projected/8237d5fa-c742-4293-802c-3fc5e29c52f0-kube-api-access-m6dnj\") pod \"8237d5fa-c742-4293-802c-3fc5e29c52f0\" (UID: \"8237d5fa-c742-4293-802c-3fc5e29c52f0\") "
Nov 03 08:15:21.478943 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.478922    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8237d5fa-c742-4293-802c-3fc5e29c52f0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8237d5fa-c742-4293-802c-3fc5e29c52f0-maas-api-service-ca\") pod \"8237d5fa-c742-4293-802c-3fc5e29c52f0\" (UID: \"8237d5fa-c742-4293-802c-3fc5e29c52f0\") "
Nov 03 08:15:21.479045 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.478960    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6cb9d88-cc98-44a7-9e72-ace1b878668f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6cb9d88-cc98-44a7-9e72-ace1b878668f-maas-api-service-ca\") pod \"b6cb9d88-cc98-44a7-9e72-ace1b878668f\" (UID: \"b6cb9d88-cc98-44a7-9e72-ace1b878668f\") "
Nov 03 08:15:21.479347 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.479313    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8237d5fa-c742-4293-802c-3fc5e29c52f0-maas-api-service-ca" pod "8237d5fa-c742-4293-802c-3fc5e29c52f0" (UID: "8237d5fa-c742-4293-802c-3fc5e29c52f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:21.479453 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.479409    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6cb9d88-cc98-44a7-9e72-ace1b878668f-maas-api-service-ca" pod "b6cb9d88-cc98-44a7-9e72-ace1b878668f" (UID: "b6cb9d88-cc98-44a7-9e72-ace1b878668f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:21.479971 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.479947    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8237d5fa-c742-4293-802c-3fc5e29c52f0-maas-api-service-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:21.480084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.479976    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6cb9d88-cc98-44a7-9e72-ace1b878668f-maas-api-service-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:21.481111 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.481086    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8237d5fa-c742-4293-802c-3fc5e29c52f0-kube-api-access-m6dnj" pod "8237d5fa-c742-4293-802c-3fc5e29c52f0" (UID: "8237d5fa-c742-4293-802c-3fc5e29c52f0"). InnerVolumeSpecName "kube-api-access-m6dnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:21.481358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.481337    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6cb9d88-cc98-44a7-9e72-ace1b878668f-kube-api-access-b8gzq" pod "b6cb9d88-cc98-44a7-9e72-ace1b878668f" (UID: "b6cb9d88-cc98-44a7-9e72-ace1b878668f"). InnerVolumeSpecName "kube-api-access-b8gzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:21.582003 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.581972    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8gzq\" (UniqueName: \"kubernetes.io/projected/b6cb9d88-cc98-44a7-9e72-ace1b878668f-kube-api-access-b8gzq\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:21.582003 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.582006    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6dnj\" (UniqueName: \"kubernetes.io/projected/8237d5fa-c742-4293-802c-3fc5e29c52f0-kube-api-access-m6dnj\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:21.840901 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.840873    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt"
Nov 03 08:15:21.841603 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.841564    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt" event={"ID":"f228239e-f1d7-4bb8-9fc9-edc630a9df41","Type":"ContainerDied","Data":"40ad98561d7e55ebeda13e6d4ce718706de5d07cf25984186857b3e607a33b15"}
Nov 03 08:15:21.841753 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.841690    5070 scope.go:122] "RemoveContainer" containerID="35a493ed5a09c6decda068878efe220c3f245a4aa3316c1b9b661a4901923be2"
Nov 03 08:15:21.845660 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.845311    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8" event={"ID":"b6cb9d88-cc98-44a7-9e72-ace1b878668f","Type":"ContainerDied","Data":"ee67c247fe2c454ce07a96993a0c15b098e6451cb33f72de3eace60c7bae4ad4"}
Nov 03 08:15:21.845660 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.845488    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8"
Nov 03 08:15:21.852189 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.852167    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f"
Nov 03 08:15:21.852265 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.852170    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f" event={"ID":"8237d5fa-c742-4293-802c-3fc5e29c52f0","Type":"ContainerDied","Data":"3a99a13fb8326caa66213ea799e97afd0d8eafc8ee74d0304f845cf76c397af5"}
Nov 03 08:15:21.854159 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.854135    5070 generic.go:356] "Generic (PLEG): container finished" podID="5a5d91d5-e0e2-41db-bea2-85a814e5274a" containerID="41c500800e162f3143ae996c45c3fca56b651f0573de978e9172dd250442d562" exitCode=60
Nov 03 08:15:21.854253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.854202    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw" event={"ID":"5a5d91d5-e0e2-41db-bea2-85a814e5274a","Type":"ContainerDied","Data":"41c500800e162f3143ae996c45c3fca56b651f0573de978e9172dd250442d562"}
Nov 03 08:15:21.854640 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.854620    5070 scope.go:122] "RemoveContainer" containerID="41c500800e162f3143ae996c45c3fca56b651f0573de978e9172dd250442d562"
Nov 03 08:15:21.854951 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:21.854927    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw_odh-ai-gateway-infra(5a5d91d5-e0e2-41db-bea2-85a814e5274a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw" podUID="5a5d91d5-e0e2-41db-bea2-85a814e5274a"
Nov 03 08:15:21.856483 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.856462    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk" event={"ID":"754211f3-c239-4469-be61-86b3bf06326c","Type":"ContainerStarted","Data":"b8fdc0996d0af4484db201c2761a1ee7eda611fdbe6784b628d0b1c0df9ddb6b"}
Nov 03 08:15:21.858674 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.858644    5070 generic.go:356] "Generic (PLEG): container finished" podID="e6086e8f-a1b6-43e2-8a92-3a962774083c" containerID="d8b35b96f49cb39821b18c039c55596293baf1e40a536a2f46556bdd64d2096f" exitCode=60
Nov 03 08:15:21.859571 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.858714    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm" event={"ID":"e6086e8f-a1b6-43e2-8a92-3a962774083c","Type":"ContainerDied","Data":"d8b35b96f49cb39821b18c039c55596293baf1e40a536a2f46556bdd64d2096f"}
Nov 03 08:15:21.859571 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.859147    5070 scope.go:122] "RemoveContainer" containerID="d8b35b96f49cb39821b18c039c55596293baf1e40a536a2f46556bdd64d2096f"
Nov 03 08:15:21.859571 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:21.859432    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm_odh-ai-gateway-infra(e6086e8f-a1b6-43e2-8a92-3a962774083c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm" podUID="e6086e8f-a1b6-43e2-8a92-3a962774083c"
Nov 03 08:15:21.862590 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.862570    5070 generic.go:356] "Generic (PLEG): container finished" podID="6200af12-2c64-4113-b61b-e754c746c706" containerID="d2d0c9d6e74ab1679342abc934bc83a121683fa2c8e7fad21399e0e5b796a950" exitCode=60
Nov 03 08:15:21.862692 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.862647    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg" event={"ID":"6200af12-2c64-4113-b61b-e754c746c706","Type":"ContainerDied","Data":"d2d0c9d6e74ab1679342abc934bc83a121683fa2c8e7fad21399e0e5b796a950"}
Nov 03 08:15:21.863141 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.863122    5070 scope.go:122] "RemoveContainer" containerID="d2d0c9d6e74ab1679342abc934bc83a121683fa2c8e7fad21399e0e5b796a950"
Nov 03 08:15:21.863375 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:21.863359    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg_odh-ai-gateway-infra(6200af12-2c64-4113-b61b-e754c746c706)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg" podUID="6200af12-2c64-4113-b61b-e754c746c706"
Nov 03 08:15:21.892395 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.892344    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8"]
Nov 03 08:15:21.897946 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.897928    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-d6a8bf-29894895-nh8h8"]
Nov 03 08:15:21.914452 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.914432    5070 scope.go:122] "RemoveContainer" containerID="b1a84e71c4a6cb9a752aa513b5341e546fb2551598d8acfb5bf51855915868d1"
Nov 03 08:15:21.925919 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.925900    5070 scope.go:122] "RemoveContainer" containerID="d79ce00f79bc52d3fcc43acd690ba21cce18ffd546f268a832d87a13f830b686"
Nov 03 08:15:21.936947 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.936917    5070 scope.go:122] "RemoveContainer" containerID="286100bfd9be06845d0d45f5bc2f54f576ab1f3ef10a12d6c0c0c53bcf0a578d"
Nov 03 08:15:21.973592 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:21.973562    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f"]
Nov 03 08:15:22.060612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.060573    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-15a633-29894895-gvj5f"]
Nov 03 08:15:22.129186 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.128273    5070 scope.go:122] "RemoveContainer" containerID="c890d0b64b747239bf8f59791c408b4e8349527e5eefbac2babf5ebe027e8d6f"
Nov 03 08:15:22.155487 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.155455    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="250bea66-7bc7-41ba-b7a4-2bc0498ed885" path="/var/lib/kubelet/pods/250bea66-7bc7-41ba-b7a4-2bc0498ed885/volumes"
Nov 03 08:15:22.156276 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.156236    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6923d57c-6c82-4fe8-bd99-15e3be887ce1" path="/var/lib/kubelet/pods/6923d57c-6c82-4fe8-bd99-15e3be887ce1/volumes"
Nov 03 08:15:22.157097 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.156864    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f9a2a18-912c-4366-94b9-96770048ec4d" path="/var/lib/kubelet/pods/6f9a2a18-912c-4366-94b9-96770048ec4d/volumes"
Nov 03 08:15:22.157996 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.157966    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d22235-c4cd-4701-a8ab-27f8143ce0fa" path="/var/lib/kubelet/pods/79d22235-c4cd-4701-a8ab-27f8143ce0fa/volumes"
Nov 03 08:15:22.158601 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.158575    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8237d5fa-c742-4293-802c-3fc5e29c52f0" path="/var/lib/kubelet/pods/8237d5fa-c742-4293-802c-3fc5e29c52f0/volumes"
Nov 03 08:15:22.163559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.159228    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86e2d036-060e-40b6-8584-bd61b2595169" path="/var/lib/kubelet/pods/86e2d036-060e-40b6-8584-bd61b2595169/volumes"
Nov 03 08:15:22.163559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.160266    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a05958e-20e4-4050-a9a8-76c22342d517" path="/var/lib/kubelet/pods/8a05958e-20e4-4050-a9a8-76c22342d517/volumes"
Nov 03 08:15:22.163559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.160861    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6cb9d88-cc98-44a7-9e72-ace1b878668f" path="/var/lib/kubelet/pods/b6cb9d88-cc98-44a7-9e72-ace1b878668f/volumes"
Nov 03 08:15:22.163559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.161432    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt"]
Nov 03 08:15:22.176262 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.176240    5070 scope.go:122] "RemoveContainer" containerID="d9222ef081bb1d0b47081201284fc577c3b45abc159df1db47e844746df60a3e"
Nov 03 08:15:22.180870 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.180739    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-e81ace-29894895-h2rwt"]
Nov 03 08:15:22.264606 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:22.264574    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:15:22.399830 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.398212    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:15:22.784930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.605418    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-workload-socket\") pod \"5bb58eee-a968-448c-9204-795ea3d444b7\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") "
Nov 03 08:15:22.784930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.605506    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bb58eee-a968-448c-9204-795ea3d444b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb58eee-a968-448c-9204-795ea3d444b7-istio-podinfo\") pod \"5bb58eee-a968-448c-9204-795ea3d444b7\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") "
Nov 03 08:15:22.784930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.605592    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-credential-socket\") pod \"5bb58eee-a968-448c-9204-795ea3d444b7\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") "
Nov 03 08:15:22.784930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.605627    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bb58eee-a968-448c-9204-795ea3d444b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb58eee-a968-448c-9204-795ea3d444b7-istiod-ca-cert\") pod \"5bb58eee-a968-448c-9204-795ea3d444b7\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") "
Nov 03 08:15:22.784930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.605864    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-istio-envoy\") pod \"5bb58eee-a968-448c-9204-795ea3d444b7\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") "
Nov 03 08:15:22.784930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.605893    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-workload-certs\") pod \"5bb58eee-a968-448c-9204-795ea3d444b7\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") "
Nov 03 08:15:22.784930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.605969    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb58eee-a968-448c-9204-795ea3d444b7-istio-token\" (UniqueName: \"kubernetes.io/projected/5bb58eee-a968-448c-9204-795ea3d444b7-istio-token\") pod \"5bb58eee-a968-448c-9204-795ea3d444b7\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") "
Nov 03 08:15:22.784930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.606035    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb58eee-a968-448c-9204-795ea3d444b7-kube-api-access-lnn5n\" (UniqueName: \"kubernetes.io/projected/5bb58eee-a968-448c-9204-795ea3d444b7-kube-api-access-lnn5n\") pod \"5bb58eee-a968-448c-9204-795ea3d444b7\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") "
Nov 03 08:15:22.784930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.606069    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-istio-data\") pod \"5bb58eee-a968-448c-9204-795ea3d444b7\" (UID: \"5bb58eee-a968-448c-9204-795ea3d444b7\") "
Nov 03 08:15:22.784930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.613865    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-workload-socket" pod "5bb58eee-a968-448c-9204-795ea3d444b7" (UID: "5bb58eee-a968-448c-9204-795ea3d444b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.784930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.619950    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb58eee-a968-448c-9204-795ea3d444b7-istiod-ca-cert" pod "5bb58eee-a968-448c-9204-795ea3d444b7" (UID: "5bb58eee-a968-448c-9204-795ea3d444b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:22.784930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.632830    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-credential-socket" pod "5bb58eee-a968-448c-9204-795ea3d444b7" (UID: "5bb58eee-a968-448c-9204-795ea3d444b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.784930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.691026    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-istio-data" pod "5bb58eee-a968-448c-9204-795ea3d444b7" (UID: "5bb58eee-a968-448c-9204-795ea3d444b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.784930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.709863    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-workload-certs" pod "5bb58eee-a968-448c-9204-795ea3d444b7" (UID: "5bb58eee-a968-448c-9204-795ea3d444b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.784930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.711531    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.784930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.711558    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.784930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.711574    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.784930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.711591    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.784930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.711607    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb58eee-a968-448c-9204-795ea3d444b7-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.784930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.752871    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:15:22.787533 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.787500    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bb58eee-a968-448c-9204-795ea3d444b7-istio-podinfo" pod "5bb58eee-a968-448c-9204-795ea3d444b7" (UID: "5bb58eee-a968-448c-9204-795ea3d444b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:15:22.787775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.787551    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb58eee-a968-448c-9204-795ea3d444b7-istio-token" pod "5bb58eee-a968-448c-9204-795ea3d444b7" (UID: "5bb58eee-a968-448c-9204-795ea3d444b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:22.787892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.787645    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-istio-envoy" pod "5bb58eee-a968-448c-9204-795ea3d444b7" (UID: "5bb58eee-a968-448c-9204-795ea3d444b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.802681 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.797714    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb58eee-a968-448c-9204-795ea3d444b7-kube-api-access-lnn5n" pod "5bb58eee-a968-448c-9204-795ea3d444b7" (UID: "5bb58eee-a968-448c-9204-795ea3d444b7"). InnerVolumeSpecName "kube-api-access-lnn5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:22.820778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.812481    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-workload-certs\") pod \"12307292-4369-4f20-9288-9b2005609416\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") "
Nov 03 08:15:22.820778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.812528    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-workload-socket\") pod \"12307292-4369-4f20-9288-9b2005609416\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") "
Nov 03 08:15:22.820778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.812579    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12307292-4369-4f20-9288-9b2005609416-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12307292-4369-4f20-9288-9b2005609416-istiod-ca-cert\") pod \"12307292-4369-4f20-9288-9b2005609416\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") "
Nov 03 08:15:22.820778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.812682    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-istio-data\") pod \"12307292-4369-4f20-9288-9b2005609416\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") "
Nov 03 08:15:22.820778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.812736    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-credential-socket\") pod \"12307292-4369-4f20-9288-9b2005609416\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") "
Nov 03 08:15:22.820778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.812817    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12307292-4369-4f20-9288-9b2005609416-istio-token\" (UniqueName: \"kubernetes.io/projected/12307292-4369-4f20-9288-9b2005609416-istio-token\") pod \"12307292-4369-4f20-9288-9b2005609416\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") "
Nov 03 08:15:22.820778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.812864    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12307292-4369-4f20-9288-9b2005609416-kube-api-access-2qkxr\" (UniqueName: \"kubernetes.io/projected/12307292-4369-4f20-9288-9b2005609416-kube-api-access-2qkxr\") pod \"12307292-4369-4f20-9288-9b2005609416\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") "
Nov 03 08:15:22.820778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.813086    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12307292-4369-4f20-9288-9b2005609416-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12307292-4369-4f20-9288-9b2005609416-istio-podinfo\") pod \"12307292-4369-4f20-9288-9b2005609416\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") "
Nov 03 08:15:22.820778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.813139    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-istio-envoy\") pod \"12307292-4369-4f20-9288-9b2005609416\" (UID: \"12307292-4369-4f20-9288-9b2005609416\") "
Nov 03 08:15:22.820778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.813983    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb58eee-a968-448c-9204-795ea3d444b7-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.820778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.814007    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb58eee-a968-448c-9204-795ea3d444b7-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.820778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.814022    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb58eee-a968-448c-9204-795ea3d444b7-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.820778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.814038    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnn5n\" (UniqueName: \"kubernetes.io/projected/5bb58eee-a968-448c-9204-795ea3d444b7-kube-api-access-lnn5n\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.820778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.814845    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-istio-data" pod "12307292-4369-4f20-9288-9b2005609416" (UID: "12307292-4369-4f20-9288-9b2005609416"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.820778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.815045    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-workload-certs" pod "12307292-4369-4f20-9288-9b2005609416" (UID: "12307292-4369-4f20-9288-9b2005609416"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.820778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.815239    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-workload-socket" pod "12307292-4369-4f20-9288-9b2005609416" (UID: "12307292-4369-4f20-9288-9b2005609416"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.820778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.815508    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12307292-4369-4f20-9288-9b2005609416-istiod-ca-cert" pod "12307292-4369-4f20-9288-9b2005609416" (UID: "12307292-4369-4f20-9288-9b2005609416"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:22.820778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.816195    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-credential-socket" pod "12307292-4369-4f20-9288-9b2005609416" (UID: "12307292-4369-4f20-9288-9b2005609416"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.820778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.819825    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-istio-envoy" pod "12307292-4369-4f20-9288-9b2005609416" (UID: "12307292-4369-4f20-9288-9b2005609416"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.822624 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.822316    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12307292-4369-4f20-9288-9b2005609416-kube-api-access-2qkxr" pod "12307292-4369-4f20-9288-9b2005609416" (UID: "12307292-4369-4f20-9288-9b2005609416"). InnerVolumeSpecName "kube-api-access-2qkxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:22.822701 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.822643    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12307292-4369-4f20-9288-9b2005609416-istio-podinfo" pod "12307292-4369-4f20-9288-9b2005609416" (UID: "12307292-4369-4f20-9288-9b2005609416"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:15:22.823814 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.823786    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12307292-4369-4f20-9288-9b2005609416-istio-token" pod "12307292-4369-4f20-9288-9b2005609416" (UID: "12307292-4369-4f20-9288-9b2005609416"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:22.853812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.853791    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:15:22.875242 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.875218    5070 generic.go:356] "Generic (PLEG): container finished" podID="5bb58eee-a968-448c-9204-795ea3d444b7" containerID="4b209588d42ea1319663a41063f7159c3050fa4cc39e16c967f02958ac542d11" exitCode=0
Nov 03 08:15:22.875378 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.875270    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"
Nov 03 08:15:22.876121 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.875320    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8" event={"ID":"5bb58eee-a968-448c-9204-795ea3d444b7","Type":"ContainerDied","Data":"4b209588d42ea1319663a41063f7159c3050fa4cc39e16c967f02958ac542d11"}
Nov 03 08:15:22.876257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.876239    5070 scope.go:122] "RemoveContainer" containerID="4b209588d42ea1319663a41063f7159c3050fa4cc39e16c967f02958ac542d11"
Nov 03 08:15:22.882271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.881684    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8" event={"ID":"5bb58eee-a968-448c-9204-795ea3d444b7","Type":"ContainerDied","Data":"e9ea20b8ac6b364f2eedb9cf9553a6b998c3be610c4340c63070c4badef159a6"}
Nov 03 08:15:22.883313 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.883163    5070 generic.go:356] "Generic (PLEG): container finished" podID="12307292-4369-4f20-9288-9b2005609416" containerID="8ebc9ee133798b1c6f28bb65926a3dad9df2da4754aebb4c84588ededc84e2c3" exitCode=0
Nov 03 08:15:22.883313 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.883211    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"
Nov 03 08:15:22.883313 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.883246    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2" event={"ID":"12307292-4369-4f20-9288-9b2005609416","Type":"ContainerDied","Data":"8ebc9ee133798b1c6f28bb65926a3dad9df2da4754aebb4c84588ededc84e2c3"}
Nov 03 08:15:22.883313 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.883298    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2" event={"ID":"12307292-4369-4f20-9288-9b2005609416","Type":"ContainerDied","Data":"e18097d08266a5336bcecf5038f162a160fffab4f7e444d44fbc079972501a0e"}
Nov 03 08:15:22.890795 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.887934    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw"]
Nov 03 08:15:22.894023 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.894003    5070 generic.go:356] "Generic (PLEG): container finished" podID="64c976ca-8c6d-4835-b5ca-ba88be627ea7" containerID="858436e67cd13dbb219a36a6c0c6514844e5c2e2eda96b78203c916c230fc665" exitCode=0
Nov 03 08:15:22.894131 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.894050    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"
Nov 03 08:15:22.894832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.894171    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr" event={"ID":"64c976ca-8c6d-4835-b5ca-ba88be627ea7","Type":"ContainerDied","Data":"858436e67cd13dbb219a36a6c0c6514844e5c2e2eda96b78203c916c230fc665"}
Nov 03 08:15:22.894832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.894215    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr" event={"ID":"64c976ca-8c6d-4835-b5ca-ba88be627ea7","Type":"ContainerDied","Data":"f714c87a8f8b50f7d562ea321ca965310fea49bc81863a9d0b97116b1ea2afdb"}
Nov 03 08:15:22.908853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.906879    5070 scope.go:122] "RemoveContainer" containerID="d48a352dce138a44dd9449506518e58e8571274844b5146526244f72ff5dbfe9"
Nov 03 08:15:22.910499 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.910473    5070 generic.go:356] "Generic (PLEG): container finished" podID="754211f3-c239-4469-be61-86b3bf06326c" containerID="b8fdc0996d0af4484db201c2761a1ee7eda611fdbe6784b628d0b1c0df9ddb6b" exitCode=60
Nov 03 08:15:22.910611 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.910594    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk" event={"ID":"754211f3-c239-4469-be61-86b3bf06326c","Type":"ContainerDied","Data":"b8fdc0996d0af4484db201c2761a1ee7eda611fdbe6784b628d0b1c0df9ddb6b"}
Nov 03 08:15:22.911418 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.911397    5070 scope.go:122] "RemoveContainer" containerID="b8fdc0996d0af4484db201c2761a1ee7eda611fdbe6784b628d0b1c0df9ddb6b"
Nov 03 08:15:22.912129 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:22.912097    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk_odh-ai-gateway-infra(754211f3-c239-4469-be61-86b3bf06326c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk" podUID="754211f3-c239-4469-be61-86b3bf06326c"
Nov 03 08:15:22.921834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.919289    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-data\") pod \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") "
Nov 03 08:15:22.921834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.919400    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64c976ca-8c6d-4835-b5ca-ba88be627ea7-kube-api-access-rgwr2\" (UniqueName: \"kubernetes.io/projected/64c976ca-8c6d-4835-b5ca-ba88be627ea7-kube-api-access-rgwr2\") pod \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") "
Nov 03 08:15:22.921834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.919464    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-podinfo\") pod \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") "
Nov 03 08:15:22.921834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.919575    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-token\" (UniqueName: \"kubernetes.io/projected/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-token\") pod \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") "
Nov 03 08:15:22.921834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.919621    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-credential-socket\") pod \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") "
Nov 03 08:15:22.921834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.919751    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-envoy\") pod \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") "
Nov 03 08:15:22.921834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.919853    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istiod-ca-cert\") pod \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") "
Nov 03 08:15:22.921834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.919917    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-workload-socket\") pod \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") "
Nov 03 08:15:22.921834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.919983    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-workload-certs\") pod \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\" (UID: \"64c976ca-8c6d-4835-b5ca-ba88be627ea7\") "
Nov 03 08:15:22.921834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.921117    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12307292-4369-4f20-9288-9b2005609416-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.921834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.921140    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.921834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.921154    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.921834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.921174    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.921834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.921187    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12307292-4369-4f20-9288-9b2005609416-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.921834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.921200    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.921834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.921214    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12307292-4369-4f20-9288-9b2005609416-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.921834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.921235    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12307292-4369-4f20-9288-9b2005609416-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.921834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.921249    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qkxr\" (UniqueName: \"kubernetes.io/projected/12307292-4369-4f20-9288-9b2005609416-kube-api-access-2qkxr\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:22.921834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.921520    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-workload-certs" pod "64c976ca-8c6d-4835-b5ca-ba88be627ea7" (UID: "64c976ca-8c6d-4835-b5ca-ba88be627ea7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.924293 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.924256    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"]
Nov 03 08:15:22.924415 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.924379    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-podinfo" pod "64c976ca-8c6d-4835-b5ca-ba88be627ea7" (UID: "64c976ca-8c6d-4835-b5ca-ba88be627ea7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:15:22.924768 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.924723    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istiod-ca-cert" pod "64c976ca-8c6d-4835-b5ca-ba88be627ea7" (UID: "64c976ca-8c6d-4835-b5ca-ba88be627ea7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:22.925070 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.925044    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-workload-socket" pod "64c976ca-8c6d-4835-b5ca-ba88be627ea7" (UID: "64c976ca-8c6d-4835-b5ca-ba88be627ea7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.925304 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.925281    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-credential-socket" pod "64c976ca-8c6d-4835-b5ca-ba88be627ea7" (UID: "64c976ca-8c6d-4835-b5ca-ba88be627ea7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.930124 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.930086    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-data" pod "64c976ca-8c6d-4835-b5ca-ba88be627ea7" (UID: "64c976ca-8c6d-4835-b5ca-ba88be627ea7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.942069 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.942046    5070 scope.go:122] "RemoveContainer" containerID="4b209588d42ea1319663a41063f7159c3050fa4cc39e16c967f02958ac542d11"
Nov 03 08:15:22.942559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.942534    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-envoy" pod "64c976ca-8c6d-4835-b5ca-ba88be627ea7" (UID: "64c976ca-8c6d-4835-b5ca-ba88be627ea7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:15:22.942696 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:22.942648    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b209588d42ea1319663a41063f7159c3050fa4cc39e16c967f02958ac542d11\": container with ID starting with 4b209588d42ea1319663a41063f7159c3050fa4cc39e16c967f02958ac542d11 not found: ID does not exist" containerID="4b209588d42ea1319663a41063f7159c3050fa4cc39e16c967f02958ac542d11"
Nov 03 08:15:22.942826 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.942802    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b209588d42ea1319663a41063f7159c3050fa4cc39e16c967f02958ac542d11"} err="failed to get container status \"4b209588d42ea1319663a41063f7159c3050fa4cc39e16c967f02958ac542d11\": rpc error: code = NotFound desc = could not find container \"4b209588d42ea1319663a41063f7159c3050fa4cc39e16c967f02958ac542d11\": container with ID starting with 4b209588d42ea1319663a41063f7159c3050fa4cc39e16c967f02958ac542d11 not found: ID does not exist"
Nov 03 08:15:22.942912 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.942902    5070 scope.go:122] "RemoveContainer" containerID="d48a352dce138a44dd9449506518e58e8571274844b5146526244f72ff5dbfe9"
Nov 03 08:15:22.943703 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:22.943669    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d48a352dce138a44dd9449506518e58e8571274844b5146526244f72ff5dbfe9\": container with ID starting with d48a352dce138a44dd9449506518e58e8571274844b5146526244f72ff5dbfe9 not found: ID does not exist" containerID="d48a352dce138a44dd9449506518e58e8571274844b5146526244f72ff5dbfe9"
Nov 03 08:15:22.944051 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.943994    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48a352dce138a44dd9449506518e58e8571274844b5146526244f72ff5dbfe9"} err="failed to get container status \"d48a352dce138a44dd9449506518e58e8571274844b5146526244f72ff5dbfe9\": rpc error: code = NotFound desc = could not find container \"d48a352dce138a44dd9449506518e58e8571274844b5146526244f72ff5dbfe9\": container with ID starting with d48a352dce138a44dd9449506518e58e8571274844b5146526244f72ff5dbfe9 not found: ID does not exist"
Nov 03 08:15:22.944119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.944057    5070 scope.go:122] "RemoveContainer" containerID="8ebc9ee133798b1c6f28bb65926a3dad9df2da4754aebb4c84588ededc84e2c3"
Nov 03 08:15:22.944807 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.944781    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c976ca-8c6d-4835-b5ca-ba88be627ea7-kube-api-access-rgwr2" pod "64c976ca-8c6d-4835-b5ca-ba88be627ea7" (UID: "64c976ca-8c6d-4835-b5ca-ba88be627ea7"). InnerVolumeSpecName "kube-api-access-rgwr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:22.944884 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.944872    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cb887217-openshift-default-6ffff5bdfc-vblb8"]
Nov 03 08:15:22.947867 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.946321    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-token" pod "64c976ca-8c6d-4835-b5ca-ba88be627ea7" (UID: "64c976ca-8c6d-4835-b5ca-ba88be627ea7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:22.953472 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.951407    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"]
Nov 03 08:15:22.958196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.958178    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-blsw2"]
Nov 03 08:15:22.961726 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.961251    5070 scope.go:122] "RemoveContainer" containerID="8ebc9ee133798b1c6f28bb65926a3dad9df2da4754aebb4c84588ededc84e2c3"
Nov 03 08:15:22.961726 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:22.961611    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ebc9ee133798b1c6f28bb65926a3dad9df2da4754aebb4c84588ededc84e2c3\": container with ID starting with 8ebc9ee133798b1c6f28bb65926a3dad9df2da4754aebb4c84588ededc84e2c3 not found: ID does not exist" containerID="8ebc9ee133798b1c6f28bb65926a3dad9df2da4754aebb4c84588ededc84e2c3"
Nov 03 08:15:22.961868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.961723    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ebc9ee133798b1c6f28bb65926a3dad9df2da4754aebb4c84588ededc84e2c3"} err="failed to get container status \"8ebc9ee133798b1c6f28bb65926a3dad9df2da4754aebb4c84588ededc84e2c3\": rpc error: code = NotFound desc = could not find container \"8ebc9ee133798b1c6f28bb65926a3dad9df2da4754aebb4c84588ededc84e2c3\": container with ID starting with 8ebc9ee133798b1c6f28bb65926a3dad9df2da4754aebb4c84588ededc84e2c3 not found: ID does not exist"
Nov 03 08:15:22.961868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.961747    5070 scope.go:122] "RemoveContainer" containerID="858436e67cd13dbb219a36a6c0c6514844e5c2e2eda96b78203c916c230fc665"
Nov 03 08:15:22.971458 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:22.971425    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm"]
Nov 03 08:15:23.007376 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.007320    5070 scope.go:122] "RemoveContainer" containerID="858436e67cd13dbb219a36a6c0c6514844e5c2e2eda96b78203c916c230fc665"
Nov 03 08:15:23.007829 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:23.007745    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858436e67cd13dbb219a36a6c0c6514844e5c2e2eda96b78203c916c230fc665\": container with ID starting with 858436e67cd13dbb219a36a6c0c6514844e5c2e2eda96b78203c916c230fc665 not found: ID does not exist" containerID="858436e67cd13dbb219a36a6c0c6514844e5c2e2eda96b78203c916c230fc665"
Nov 03 08:15:23.008146 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.008115    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858436e67cd13dbb219a36a6c0c6514844e5c2e2eda96b78203c916c230fc665"} err="failed to get container status \"858436e67cd13dbb219a36a6c0c6514844e5c2e2eda96b78203c916c230fc665\": rpc error: code = NotFound desc = could not find container \"858436e67cd13dbb219a36a6c0c6514844e5c2e2eda96b78203c916c230fc665\": container with ID starting with 858436e67cd13dbb219a36a6c0c6514844e5c2e2eda96b78203c916c230fc665 not found: ID does not exist"
Nov 03 08:15:23.008281 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.008148    5070 scope.go:122] "RemoveContainer" containerID="53a63b2b0b2df07aa52a6e9e97bb34e213531c7245f6f01f11fa89d1a7216310"
Nov 03 08:15:23.024411 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.024378    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:23.024411 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.024413    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:23.024597 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.024429    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:23.024597 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.024444    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:23.024597 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.024458    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:23.024597 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.024472    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgwr2\" (UniqueName: \"kubernetes.io/projected/64c976ca-8c6d-4835-b5ca-ba88be627ea7-kube-api-access-rgwr2\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:23.024597 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.024487    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:23.024597 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.024501    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c976ca-8c6d-4835-b5ca-ba88be627ea7-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:23.024597 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.024514    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c976ca-8c6d-4835-b5ca-ba88be627ea7-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:23.092734 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.092704    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk"]
Nov 03 08:15:23.135963 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.135931    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg"]
Nov 03 08:15:23.192719 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.192689    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw"
Nov 03 08:15:23.228974 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.227135    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a5d91d5-e0e2-41db-bea2-85a814e5274a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a5d91d5-e0e2-41db-bea2-85a814e5274a-maas-api-service-ca\") pod \"5a5d91d5-e0e2-41db-bea2-85a814e5274a\" (UID: \"5a5d91d5-e0e2-41db-bea2-85a814e5274a\") "
Nov 03 08:15:23.228974 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.227192    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a5d91d5-e0e2-41db-bea2-85a814e5274a-kube-api-access-jmvrg\" (UniqueName: \"kubernetes.io/projected/5a5d91d5-e0e2-41db-bea2-85a814e5274a-kube-api-access-jmvrg\") pod \"5a5d91d5-e0e2-41db-bea2-85a814e5274a\" (UID: \"5a5d91d5-e0e2-41db-bea2-85a814e5274a\") "
Nov 03 08:15:23.228974 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.227605    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a5d91d5-e0e2-41db-bea2-85a814e5274a-maas-api-service-ca" pod "5a5d91d5-e0e2-41db-bea2-85a814e5274a" (UID: "5a5d91d5-e0e2-41db-bea2-85a814e5274a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:23.228974 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.228003    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a5d91d5-e0e2-41db-bea2-85a814e5274a-maas-api-service-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:23.230265 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.230225    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5d91d5-e0e2-41db-bea2-85a814e5274a-kube-api-access-jmvrg" pod "5a5d91d5-e0e2-41db-bea2-85a814e5274a" (UID: "5a5d91d5-e0e2-41db-bea2-85a814e5274a"). InnerVolumeSpecName "kube-api-access-jmvrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:23.268352 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.268332    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm"
Nov 03 08:15:23.279275 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.279250    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"]
Nov 03 08:15:23.291577 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.291545    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-hl9gr"]
Nov 03 08:15:23.329421 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.329340    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6086e8f-a1b6-43e2-8a92-3a962774083c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6086e8f-a1b6-43e2-8a92-3a962774083c-maas-api-service-ca\") pod \"e6086e8f-a1b6-43e2-8a92-3a962774083c\" (UID: \"e6086e8f-a1b6-43e2-8a92-3a962774083c\") "
Nov 03 08:15:23.329629 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.329612    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6086e8f-a1b6-43e2-8a92-3a962774083c-kube-api-access-9sg8p\" (UniqueName: \"kubernetes.io/projected/e6086e8f-a1b6-43e2-8a92-3a962774083c-kube-api-access-9sg8p\") pod \"e6086e8f-a1b6-43e2-8a92-3a962774083c\" (UID: \"e6086e8f-a1b6-43e2-8a92-3a962774083c\") "
Nov 03 08:15:23.329710 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.329676    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6086e8f-a1b6-43e2-8a92-3a962774083c-maas-api-service-ca" pod "e6086e8f-a1b6-43e2-8a92-3a962774083c" (UID: "e6086e8f-a1b6-43e2-8a92-3a962774083c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:23.330446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.330418    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmvrg\" (UniqueName: \"kubernetes.io/projected/5a5d91d5-e0e2-41db-bea2-85a814e5274a-kube-api-access-jmvrg\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:23.330446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.330447    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6086e8f-a1b6-43e2-8a92-3a962774083c-maas-api-service-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:23.332194 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.332166    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6086e8f-a1b6-43e2-8a92-3a962774083c-kube-api-access-9sg8p" pod "e6086e8f-a1b6-43e2-8a92-3a962774083c" (UID: "e6086e8f-a1b6-43e2-8a92-3a962774083c"). InnerVolumeSpecName "kube-api-access-9sg8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:23.361944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.361924    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg"
Nov 03 08:15:23.431504 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.431458    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6200af12-2c64-4113-b61b-e754c746c706-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6200af12-2c64-4113-b61b-e754c746c706-maas-api-service-ca\") pod \"6200af12-2c64-4113-b61b-e754c746c706\" (UID: \"6200af12-2c64-4113-b61b-e754c746c706\") "
Nov 03 08:15:23.431751 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.431553    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6200af12-2c64-4113-b61b-e754c746c706-kube-api-access-4qkpg\" (UniqueName: \"kubernetes.io/projected/6200af12-2c64-4113-b61b-e754c746c706-kube-api-access-4qkpg\") pod \"6200af12-2c64-4113-b61b-e754c746c706\" (UID: \"6200af12-2c64-4113-b61b-e754c746c706\") "
Nov 03 08:15:23.432237 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.432202    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6200af12-2c64-4113-b61b-e754c746c706-maas-api-service-ca" pod "6200af12-2c64-4113-b61b-e754c746c706" (UID: "6200af12-2c64-4113-b61b-e754c746c706"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:23.432637 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.432607    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sg8p\" (UniqueName: \"kubernetes.io/projected/e6086e8f-a1b6-43e2-8a92-3a962774083c-kube-api-access-9sg8p\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:23.432836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.432641    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6200af12-2c64-4113-b61b-e754c746c706-maas-api-service-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:23.434466 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.434445    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6200af12-2c64-4113-b61b-e754c746c706-kube-api-access-4qkpg" pod "6200af12-2c64-4113-b61b-e754c746c706" (UID: "6200af12-2c64-4113-b61b-e754c746c706"). InnerVolumeSpecName "kube-api-access-4qkpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:23.533846 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.533820    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qkpg\" (UniqueName: \"kubernetes.io/projected/6200af12-2c64-4113-b61b-e754c746c706-kube-api-access-4qkpg\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:23.983493 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.983457    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw" event={"ID":"5a5d91d5-e0e2-41db-bea2-85a814e5274a","Type":"ContainerDied","Data":"b86265e065da482028773256d21b071c09a239929035669664c08266fedb4fe7"}
Nov 03 08:15:23.983791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.983771    5070 scope.go:122] "RemoveContainer" containerID="41c500800e162f3143ae996c45c3fca56b651f0573de978e9172dd250442d562"
Nov 03 08:15:23.984016 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:23.983992    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw"
Nov 03 08:15:24.004446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.004417    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm" event={"ID":"e6086e8f-a1b6-43e2-8a92-3a962774083c","Type":"ContainerDied","Data":"3e142f2d35ad8dd7b29ba715e935b89880b48a8ab23ef96b28a14166fa448c1a"}
Nov 03 08:15:24.004714 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.004695    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm"
Nov 03 08:15:24.056186 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.049046    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg"
Nov 03 08:15:24.085057 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.085024    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12307292-4369-4f20-9288-9b2005609416" path="/var/lib/kubelet/pods/12307292-4369-4f20-9288-9b2005609416/volumes"
Nov 03 08:15:24.088070 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.088040    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bb58eee-a968-448c-9204-795ea3d444b7" path="/var/lib/kubelet/pods/5bb58eee-a968-448c-9204-795ea3d444b7/volumes"
Nov 03 08:15:24.089361 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.089334    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64c976ca-8c6d-4835-b5ca-ba88be627ea7" path="/var/lib/kubelet/pods/64c976ca-8c6d-4835-b5ca-ba88be627ea7/volumes"
Nov 03 08:15:24.091430 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.091411    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f228239e-f1d7-4bb8-9fc9-edc630a9df41" path="/var/lib/kubelet/pods/f228239e-f1d7-4bb8-9fc9-edc630a9df41/volumes"
Nov 03 08:15:24.159147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.143667    5070 scope.go:122] "RemoveContainer" containerID="d8b35b96f49cb39821b18c039c55596293baf1e40a536a2f46556bdd64d2096f"
Nov 03 08:15:24.159147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.147414    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg" event={"ID":"6200af12-2c64-4113-b61b-e754c746c706","Type":"ContainerDied","Data":"b4c10b6160fde13c21dc318dcd8289bea5dde126a8b9ca3be770066806609305"}
Nov 03 08:15:24.183900 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.183035    5070 scope.go:122] "RemoveContainer" containerID="d2d0c9d6e74ab1679342abc934bc83a121683fa2c8e7fad21399e0e5b796a950"
Nov 03 08:15:24.183900 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.183241    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg"]
Nov 03 08:15:24.188744 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.188708    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-178e68-29894895-kkxvg"]
Nov 03 08:15:24.196569 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.196539    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw"]
Nov 03 08:15:24.200942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.200917    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-76ef13-29894895-mqdrw"]
Nov 03 08:15:24.206972 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.206948    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm"]
Nov 03 08:15:24.213559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.213537    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-7a6481-29894895-nf7nm"]
Nov 03 08:15:24.431874 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.431837    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk"
Nov 03 08:15:24.515963 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.515881    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/754211f3-c239-4469-be61-86b3bf06326c-kube-api-access-ngjs5\" (UniqueName: \"kubernetes.io/projected/754211f3-c239-4469-be61-86b3bf06326c-kube-api-access-ngjs5\") pod \"754211f3-c239-4469-be61-86b3bf06326c\" (UID: \"754211f3-c239-4469-be61-86b3bf06326c\") "
Nov 03 08:15:24.516178 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.516159    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/754211f3-c239-4469-be61-86b3bf06326c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/754211f3-c239-4469-be61-86b3bf06326c-maas-api-service-ca\") pod \"754211f3-c239-4469-be61-86b3bf06326c\" (UID: \"754211f3-c239-4469-be61-86b3bf06326c\") "
Nov 03 08:15:24.516568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.516541    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/754211f3-c239-4469-be61-86b3bf06326c-maas-api-service-ca" pod "754211f3-c239-4469-be61-86b3bf06326c" (UID: "754211f3-c239-4469-be61-86b3bf06326c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:24.516894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.516875    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/754211f3-c239-4469-be61-86b3bf06326c-maas-api-service-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:24.518069 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.518043    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754211f3-c239-4469-be61-86b3bf06326c-kube-api-access-ngjs5" pod "754211f3-c239-4469-be61-86b3bf06326c" (UID: "754211f3-c239-4469-be61-86b3bf06326c"). InnerVolumeSpecName "kube-api-access-ngjs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:24.617728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.617699    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngjs5\" (UniqueName: \"kubernetes.io/projected/754211f3-c239-4469-be61-86b3bf06326c-kube-api-access-ngjs5\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:24.825174 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.825105    5070 scope.go:122] "RemoveContainer" containerID="b82fb17dca49c9a22e3d4cb4d1af5726eca1040a410f06f6e30ef8f01d029ac4"
Nov 03 08:15:24.836297 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.836271    5070 scope.go:122] "RemoveContainer" containerID="0024023d85a82fa0aec464eb3c50ae65ae41a01be8a38dcd0bb643a2e157dcff"
Nov 03 08:15:24.847118 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.847095    5070 scope.go:122] "RemoveContainer" containerID="0b6bf82e878d3da781dac8d1096c51546c81a109f7913262b3016a94b6d1d049"
Nov 03 08:15:24.856967 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:24.856946    5070 scope.go:122] "RemoveContainer" containerID="d0bb89b21273c681c2a9cb8d55c362a333f6e42ac96a8a0b5485aa9aa1202529"
Nov 03 08:15:25.074101 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:25.074078    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk"
Nov 03 08:15:25.074202 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:25.074117    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk" event={"ID":"754211f3-c239-4469-be61-86b3bf06326c","Type":"ContainerDied","Data":"582f3fc3ea2b696908b60b496184ca7d33c179f293d4244f71d8005e2c2e6e7e"}
Nov 03 08:15:25.074202 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:25.074190    5070 scope.go:122] "RemoveContainer" containerID="b8fdc0996d0af4484db201c2761a1ee7eda611fdbe6784b628d0b1c0df9ddb6b"
Nov 03 08:15:25.099007 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:25.098984    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk"]
Nov 03 08:15:25.103155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:25.103133    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a60004-29894895-hjgmk"]
Nov 03 08:15:26.028026 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:26.027993    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a5d91d5-e0e2-41db-bea2-85a814e5274a" path="/var/lib/kubelet/pods/5a5d91d5-e0e2-41db-bea2-85a814e5274a/volumes"
Nov 03 08:15:26.028530 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:26.028515    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6200af12-2c64-4113-b61b-e754c746c706" path="/var/lib/kubelet/pods/6200af12-2c64-4113-b61b-e754c746c706/volumes"
Nov 03 08:15:26.028950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:26.028936    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="754211f3-c239-4469-be61-86b3bf06326c" path="/var/lib/kubelet/pods/754211f3-c239-4469-be61-86b3bf06326c/volumes"
Nov 03 08:15:26.029553 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:26.029537    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6086e8f-a1b6-43e2-8a92-3a962774083c" path="/var/lib/kubelet/pods/e6086e8f-a1b6-43e2-8a92-3a962774083c/volumes"
Nov 03 08:15:29.371101 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.371059    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"]
Nov 03 08:15:29.372210 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372185    5070 cpu_manager.go:414] "containerMap: removing container" podUID="6f9a2a18-912c-4366-94b9-96770048ec4d" containerName="cleanup"
Nov 03 08:15:29.372307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372212    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f9a2a18-912c-4366-94b9-96770048ec4d" containerName="cleanup"
Nov 03 08:15:29.372307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372231    5070 cpu_manager.go:414] "containerMap: removing container" podUID="79d22235-c4cd-4701-a8ab-27f8143ce0fa" containerName="cleanup"
Nov 03 08:15:29.372307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372239    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d22235-c4cd-4701-a8ab-27f8143ce0fa" containerName="cleanup"
Nov 03 08:15:29.372307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372248    5070 cpu_manager.go:414] "containerMap: removing container" podUID="8237d5fa-c742-4293-802c-3fc5e29c52f0" containerName="cleanup"
Nov 03 08:15:29.372307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372257    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8237d5fa-c742-4293-802c-3fc5e29c52f0" containerName="cleanup"
Nov 03 08:15:29.372307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372272    5070 cpu_manager.go:414] "containerMap: removing container" podUID="f228239e-f1d7-4bb8-9fc9-edc630a9df41" containerName="cleanup"
Nov 03 08:15:29.372307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372279    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f228239e-f1d7-4bb8-9fc9-edc630a9df41" containerName="cleanup"
Nov 03 08:15:29.372307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372289    5070 cpu_manager.go:414] "containerMap: removing container" podUID="f228239e-f1d7-4bb8-9fc9-edc630a9df41" containerName="cleanup"
Nov 03 08:15:29.372307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372297    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f228239e-f1d7-4bb8-9fc9-edc630a9df41" containerName="cleanup"
Nov 03 08:15:29.372307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372311    5070 cpu_manager.go:414] "containerMap: removing container" podUID="8a05958e-20e4-4050-a9a8-76c22342d517" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372318    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a05958e-20e4-4050-a9a8-76c22342d517" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372337    5070 cpu_manager.go:414] "containerMap: removing container" podUID="250bea66-7bc7-41ba-b7a4-2bc0498ed885" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372344    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="250bea66-7bc7-41ba-b7a4-2bc0498ed885" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372355    5070 cpu_manager.go:414] "containerMap: removing container" podUID="5a5d91d5-e0e2-41db-bea2-85a814e5274a" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372361    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a5d91d5-e0e2-41db-bea2-85a814e5274a" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372429    5070 cpu_manager.go:414] "containerMap: removing container" podUID="6923d57c-6c82-4fe8-bd99-15e3be887ce1" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372439    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6923d57c-6c82-4fe8-bd99-15e3be887ce1" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372455    5070 cpu_manager.go:414] "containerMap: removing container" podUID="8a05958e-20e4-4050-a9a8-76c22342d517" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372463    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a05958e-20e4-4050-a9a8-76c22342d517" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372477    5070 cpu_manager.go:414] "containerMap: removing container" podUID="5bb58eee-a968-448c-9204-795ea3d444b7" containerName="istio-proxy"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372484    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb58eee-a968-448c-9204-795ea3d444b7" containerName="istio-proxy"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372497    5070 cpu_manager.go:414] "containerMap: removing container" podUID="6f9a2a18-912c-4366-94b9-96770048ec4d" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372504    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f9a2a18-912c-4366-94b9-96770048ec4d" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372518    5070 cpu_manager.go:414] "containerMap: removing container" podUID="b6cb9d88-cc98-44a7-9e72-ace1b878668f" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372525    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6cb9d88-cc98-44a7-9e72-ace1b878668f" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372538    5070 cpu_manager.go:414] "containerMap: removing container" podUID="b6cb9d88-cc98-44a7-9e72-ace1b878668f" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372544    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6cb9d88-cc98-44a7-9e72-ace1b878668f" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372560    5070 cpu_manager.go:414] "containerMap: removing container" podUID="64c976ca-8c6d-4835-b5ca-ba88be627ea7" containerName="istio-proxy"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372567    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64c976ca-8c6d-4835-b5ca-ba88be627ea7" containerName="istio-proxy"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372579    5070 cpu_manager.go:414] "containerMap: removing container" podUID="e6086e8f-a1b6-43e2-8a92-3a962774083c" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372585    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6086e8f-a1b6-43e2-8a92-3a962774083c" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372597    5070 cpu_manager.go:414] "containerMap: removing container" podUID="754211f3-c239-4469-be61-86b3bf06326c" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372604    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="754211f3-c239-4469-be61-86b3bf06326c" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372624    5070 cpu_manager.go:414] "containerMap: removing container" podUID="754211f3-c239-4469-be61-86b3bf06326c" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372631    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="754211f3-c239-4469-be61-86b3bf06326c" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372640    5070 cpu_manager.go:414] "containerMap: removing container" podUID="e6086e8f-a1b6-43e2-8a92-3a962774083c" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372647    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6086e8f-a1b6-43e2-8a92-3a962774083c" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372662    5070 cpu_manager.go:414] "containerMap: removing container" podUID="86e2d036-060e-40b6-8584-bd61b2595169" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372669    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86e2d036-060e-40b6-8584-bd61b2595169" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372680    5070 cpu_manager.go:414] "containerMap: removing container" podUID="6200af12-2c64-4113-b61b-e754c746c706" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372687    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6200af12-2c64-4113-b61b-e754c746c706" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372696    5070 cpu_manager.go:414] "containerMap: removing container" podUID="86e2d036-060e-40b6-8584-bd61b2595169" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372703    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86e2d036-060e-40b6-8584-bd61b2595169" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372717    5070 cpu_manager.go:414] "containerMap: removing container" podUID="e6086e8f-a1b6-43e2-8a92-3a962774083c" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372724    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6086e8f-a1b6-43e2-8a92-3a962774083c" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372739    5070 cpu_manager.go:414] "containerMap: removing container" podUID="6200af12-2c64-4113-b61b-e754c746c706" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372747    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6200af12-2c64-4113-b61b-e754c746c706" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372774    5070 cpu_manager.go:414] "containerMap: removing container" podUID="754211f3-c239-4469-be61-86b3bf06326c" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372781    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="754211f3-c239-4469-be61-86b3bf06326c" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372797    5070 cpu_manager.go:414] "containerMap: removing container" podUID="5a5d91d5-e0e2-41db-bea2-85a814e5274a" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372804    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a5d91d5-e0e2-41db-bea2-85a814e5274a" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372821    5070 cpu_manager.go:414] "containerMap: removing container" podUID="8237d5fa-c742-4293-802c-3fc5e29c52f0" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372829    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8237d5fa-c742-4293-802c-3fc5e29c52f0" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372842    5070 cpu_manager.go:414] "containerMap: removing container" podUID="86e2d036-060e-40b6-8584-bd61b2595169" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372849    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86e2d036-060e-40b6-8584-bd61b2595169" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372865    5070 cpu_manager.go:414] "containerMap: removing container" podUID="79d22235-c4cd-4701-a8ab-27f8143ce0fa" containerName="cleanup"
Nov 03 08:15:29.372847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372873    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d22235-c4cd-4701-a8ab-27f8143ce0fa" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372888    5070 cpu_manager.go:414] "containerMap: removing container" podUID="250bea66-7bc7-41ba-b7a4-2bc0498ed885" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372895    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="250bea66-7bc7-41ba-b7a4-2bc0498ed885" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372911    5070 cpu_manager.go:414] "containerMap: removing container" podUID="12307292-4369-4f20-9288-9b2005609416" containerName="istio-proxy"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372918    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12307292-4369-4f20-9288-9b2005609416" containerName="istio-proxy"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372931    5070 cpu_manager.go:414] "containerMap: removing container" podUID="6200af12-2c64-4113-b61b-e754c746c706" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372938    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6200af12-2c64-4113-b61b-e754c746c706" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372946    5070 cpu_manager.go:414] "containerMap: removing container" podUID="6923d57c-6c82-4fe8-bd99-15e3be887ce1" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.372953    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6923d57c-6c82-4fe8-bd99-15e3be887ce1" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373133    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f9a2a18-912c-4366-94b9-96770048ec4d" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373148    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d22235-c4cd-4701-a8ab-27f8143ce0fa" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373158    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="8237d5fa-c742-4293-802c-3fc5e29c52f0" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373169    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="f228239e-f1d7-4bb8-9fc9-edc630a9df41" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373179    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="f228239e-f1d7-4bb8-9fc9-edc630a9df41" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373190    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a05958e-20e4-4050-a9a8-76c22342d517" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373198    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="8237d5fa-c742-4293-802c-3fc5e29c52f0" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373207    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="250bea66-7bc7-41ba-b7a4-2bc0498ed885" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373216    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a5d91d5-e0e2-41db-bea2-85a814e5274a" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373226    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="6923d57c-6c82-4fe8-bd99-15e3be887ce1" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373235    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb58eee-a968-448c-9204-795ea3d444b7" containerName="istio-proxy"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373243    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f9a2a18-912c-4366-94b9-96770048ec4d" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373254    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6cb9d88-cc98-44a7-9e72-ace1b878668f" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373263    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6cb9d88-cc98-44a7-9e72-ace1b878668f" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373274    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="64c976ca-8c6d-4835-b5ca-ba88be627ea7" containerName="istio-proxy"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373282    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6086e8f-a1b6-43e2-8a92-3a962774083c" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373290    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="754211f3-c239-4469-be61-86b3bf06326c" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373299    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a5d91d5-e0e2-41db-bea2-85a814e5274a" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373309    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="754211f3-c239-4469-be61-86b3bf06326c" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373316    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6086e8f-a1b6-43e2-8a92-3a962774083c" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373325    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="86e2d036-060e-40b6-8584-bd61b2595169" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373334    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="6923d57c-6c82-4fe8-bd99-15e3be887ce1" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373342    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="6200af12-2c64-4113-b61b-e754c746c706" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373350    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="86e2d036-060e-40b6-8584-bd61b2595169" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373359    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6086e8f-a1b6-43e2-8a92-3a962774083c" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373368    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="6200af12-2c64-4113-b61b-e754c746c706" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373375    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="754211f3-c239-4469-be61-86b3bf06326c" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373385    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a05958e-20e4-4050-a9a8-76c22342d517" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373394    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="86e2d036-060e-40b6-8584-bd61b2595169" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373406    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d22235-c4cd-4701-a8ab-27f8143ce0fa" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373414    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="250bea66-7bc7-41ba-b7a4-2bc0498ed885" containerName="cleanup"
Nov 03 08:15:29.376139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.373422    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="12307292-4369-4f20-9288-9b2005609416" containerName="istio-proxy"
Nov 03 08:15:29.388388 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.388361    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.395794 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.394747    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"]
Nov 03 08:15:29.395794 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.394947    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-27c674-openshift-default-dockercfg-v92fc\""
Nov 03 08:15:29.517246 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.517215    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istiod-ca-cert\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.517411 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.517269    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-workload-socket\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.517411 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.517361    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-credential-socket\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.517411 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.517397    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-data\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.517554 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.517426    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-podinfo\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.517554 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.517544    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-envoy\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.517715 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.517694    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-workload-certs\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.517843 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.517812    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-token\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.517923 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.517855    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddb2t\" (UniqueName: \"kubernetes.io/projected/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-kube-api-access-ddb2t\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.625124 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.624264    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-workload-certs\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.625439 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.625053    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-workload-certs\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.625582 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.625563    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-token\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.625723 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.625707    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddb2t\" (UniqueName: \"kubernetes.io/projected/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-kube-api-access-ddb2t\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.626001 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.625971    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istiod-ca-cert\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.636145 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.636109    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-workload-socket\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.636391 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.636374    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-credential-socket\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.636539 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.636519    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-data\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.636711 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.636685    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-podinfo\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.660199 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.646946    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-envoy\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.660199 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.637274    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-workload-socket\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.660199 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.637479    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-data\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.660199 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.626900    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istiod-ca-cert\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.660199 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.637056    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-credential-socket\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.660199 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.655161    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-envoy\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.660199 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.655279    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-podinfo\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.660199 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.655586    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-token\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.681837 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.678709    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddb2t\" (UniqueName: \"kubernetes.io/projected/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-kube-api-access-ddb2t\") pod \"e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") " pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:29.722571 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:29.722538    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:30.197013 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:30.196982    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"]
Nov 03 08:15:30.197013 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:15:30.197001    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3803b1_7a4f_4e73_b541_9cb0c27c624d.slice/crio-778e63d0bb7052b080e429cd50344bddc679612b183e6f3cb2d517ad4fca7e54 WatchSource:0}: Error finding container 778e63d0bb7052b080e429cd50344bddc679612b183e6f3cb2d517ad4fca7e54: Status 404 returned error can't find the container with id 778e63d0bb7052b080e429cd50344bddc679612b183e6f3cb2d517ad4fca7e54
Nov 03 08:15:30.200316 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:30.200279    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:15:30.200423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:30.200348    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:15:30.200423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:30.200377    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:15:31.138194 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:31.138149    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws" event={"ID":"5d3803b1-7a4f-4e73-b541-9cb0c27c624d","Type":"ContainerStarted","Data":"b7d9e635028bca7e1d91ffdc288d9cbb50c6df1cad4b6277f7bc44e53d897936"}
Nov 03 08:15:31.138799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:31.138224    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws" event={"ID":"5d3803b1-7a4f-4e73-b541-9cb0c27c624d","Type":"ContainerStarted","Data":"778e63d0bb7052b080e429cd50344bddc679612b183e6f3cb2d517ad4fca7e54"}
Nov 03 08:15:31.158280 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:31.158231    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws" podStartSLOduration=2.158215889 podStartE2EDuration="2.158215889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:31.155031041 +0000 UTC m=+1329.269494975" watchObservedRunningTime="2026-11-03 08:15:31.158215889 +0000 UTC m=+1329.272679820"
Nov 03 08:15:31.734789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:31.733130    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:31.763778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:31.763113    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:32.185784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:32.183068    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:32.188935 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:32.188908    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:15:34.672430 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.672398    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"]
Nov 03 08:15:34.674092 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.674064    5070 cpu_manager.go:414] "containerMap: removing container" podUID="5a5d91d5-e0e2-41db-bea2-85a814e5274a" containerName="cleanup"
Nov 03 08:15:34.674266 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.674250    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a5d91d5-e0e2-41db-bea2-85a814e5274a" containerName="cleanup"
Nov 03 08:15:34.674390 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.674376    5070 cpu_manager.go:414] "containerMap: removing container" podUID="8a05958e-20e4-4050-a9a8-76c22342d517" containerName="cleanup"
Nov 03 08:15:34.674486 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.674474    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a05958e-20e4-4050-a9a8-76c22342d517" containerName="cleanup"
Nov 03 08:15:34.674580 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.674569    5070 cpu_manager.go:414] "containerMap: removing container" podUID="5bb58eee-a968-448c-9204-795ea3d444b7" containerName="istio-proxy"
Nov 03 08:15:34.674665 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.674654    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb58eee-a968-448c-9204-795ea3d444b7" containerName="istio-proxy"
Nov 03 08:15:34.674752 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.674740    5070 cpu_manager.go:414] "containerMap: removing container" podUID="6f9a2a18-912c-4366-94b9-96770048ec4d" containerName="cleanup"
Nov 03 08:15:34.674889 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.674875    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f9a2a18-912c-4366-94b9-96770048ec4d" containerName="cleanup"
Nov 03 08:15:34.674985 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.674974    5070 cpu_manager.go:414] "containerMap: removing container" podUID="b6cb9d88-cc98-44a7-9e72-ace1b878668f" containerName="cleanup"
Nov 03 08:15:34.675070 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.675059    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6cb9d88-cc98-44a7-9e72-ace1b878668f" containerName="cleanup"
Nov 03 08:15:34.675217 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.675205    5070 cpu_manager.go:414] "containerMap: removing container" podUID="250bea66-7bc7-41ba-b7a4-2bc0498ed885" containerName="cleanup"
Nov 03 08:15:34.675307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.675294    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="250bea66-7bc7-41ba-b7a4-2bc0498ed885" containerName="cleanup"
Nov 03 08:15:34.675410 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.675398    5070 cpu_manager.go:414] "containerMap: removing container" podUID="6923d57c-6c82-4fe8-bd99-15e3be887ce1" containerName="cleanup"
Nov 03 08:15:34.675492 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.675481    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6923d57c-6c82-4fe8-bd99-15e3be887ce1" containerName="cleanup"
Nov 03 08:15:34.675587 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.675577    5070 cpu_manager.go:414] "containerMap: removing container" podUID="79d22235-c4cd-4701-a8ab-27f8143ce0fa" containerName="cleanup"
Nov 03 08:15:34.675666 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.675656    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d22235-c4cd-4701-a8ab-27f8143ce0fa" containerName="cleanup"
Nov 03 08:15:34.675769 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.675743    5070 cpu_manager.go:414] "containerMap: removing container" podUID="8237d5fa-c742-4293-802c-3fc5e29c52f0" containerName="cleanup"
Nov 03 08:15:34.675871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.675858    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8237d5fa-c742-4293-802c-3fc5e29c52f0" containerName="cleanup"
Nov 03 08:15:34.675966 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.675955    5070 cpu_manager.go:414] "containerMap: removing container" podUID="f228239e-f1d7-4bb8-9fc9-edc630a9df41" containerName="cleanup"
Nov 03 08:15:34.676052 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.676041    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f228239e-f1d7-4bb8-9fc9-edc630a9df41" containerName="cleanup"
Nov 03 08:15:34.676314 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.676296    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="f228239e-f1d7-4bb8-9fc9-edc630a9df41" containerName="cleanup"
Nov 03 08:15:34.676423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.676412    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="8237d5fa-c742-4293-802c-3fc5e29c52f0" containerName="cleanup"
Nov 03 08:15:34.676524 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.676513    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="250bea66-7bc7-41ba-b7a4-2bc0498ed885" containerName="cleanup"
Nov 03 08:15:34.676614 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.676604    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb58eee-a968-448c-9204-795ea3d444b7" containerName="istio-proxy"
Nov 03 08:15:34.676706 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.676696    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6cb9d88-cc98-44a7-9e72-ace1b878668f" containerName="cleanup"
Nov 03 08:15:34.676811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.676799    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a5d91d5-e0e2-41db-bea2-85a814e5274a" containerName="cleanup"
Nov 03 08:15:34.676917 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.676907    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="6923d57c-6c82-4fe8-bd99-15e3be887ce1" containerName="cleanup"
Nov 03 08:15:34.677003 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.676992    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="6200af12-2c64-4113-b61b-e754c746c706" containerName="cleanup"
Nov 03 08:15:34.677094 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.677084    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a05958e-20e4-4050-a9a8-76c22342d517" containerName="cleanup"
Nov 03 08:15:34.677183 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.677173    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f9a2a18-912c-4366-94b9-96770048ec4d" containerName="cleanup"
Nov 03 08:15:34.677265 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.677256    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d22235-c4cd-4701-a8ab-27f8143ce0fa" containerName="cleanup"
Nov 03 08:15:34.682879 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.682109    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.687049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.687025    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"]
Nov 03 08:15:34.702879 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.702853    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b8a152-openshift-default-dockercfg-pksss\""
Nov 03 08:15:34.705383 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.705349    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istiod-ca-cert\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.705506 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.705403    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-token\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.705563 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.705542    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-podinfo\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.705660 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.705635    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hg2s\" (UniqueName: \"kubernetes.io/projected/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-kube-api-access-9hg2s\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.705835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.705795    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-workload-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.705835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.705829    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-envoy\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.705965 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.705858    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-workload-certs\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.705965 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.705939    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-data\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.706061 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.705993    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-credential-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.807645 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.807609    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istiod-ca-cert\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.807645 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.807647    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-token\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.807935 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.807680    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-podinfo\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.807935 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.807738    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hg2s\" (UniqueName: \"kubernetes.io/projected/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-kube-api-access-9hg2s\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.807935 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.807835    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-workload-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.807935 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.807863    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-envoy\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.807935 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.807888    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-workload-certs\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.808283 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.807979    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-data\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.808283 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.808032    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-credential-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.809579 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.809524    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-workload-certs\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.810009 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.809952    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-credential-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.810009 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.809980    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-workload-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.810236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.810205    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istiod-ca-cert\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.810574 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.810547    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-data\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.812409 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.812366    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-podinfo\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.817458 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.817433    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-envoy\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.819487 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.819462    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-token\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:34.819883 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:34.819786    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hg2s\" (UniqueName: \"kubernetes.io/projected/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-kube-api-access-9hg2s\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:35.156349 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:35.156315    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:35.657255 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:35.656942    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"]
Nov 03 08:15:35.661418 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:35.661372    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:15:35.661529 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:35.661470    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:15:35.661529 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:35.661511    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:15:36.506831 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:36.504387    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb" event={"ID":"d915ac09-a22a-4e5d-943a-97cdeb2d9c24","Type":"ContainerStarted","Data":"38969c7bdd9b825821042634a5aac777ff416473f595f646a7ca0043dac7cd4a"}
Nov 03 08:15:36.506831 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:36.504459    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb" event={"ID":"d915ac09-a22a-4e5d-943a-97cdeb2d9c24","Type":"ContainerStarted","Data":"9400476ef3dd3ad4541ed7f2b386b3de41dd986bd900039a53bb9da225960a93"}
Nov 03 08:15:36.589783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:36.587457    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb" podStartSLOduration=2.587436856 podStartE2EDuration="2.587436856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:36.584353376 +0000 UTC m=+1334.698817309" watchObservedRunningTime="2026-11-03 08:15:36.587436856 +0000 UTC m=+1334.701900807"
Nov 03 08:15:36.659685 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:36.657959    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt"]
Nov 03 08:15:36.659933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:36.659797    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-94e3cb-xlkjt"]
Nov 03 08:15:37.157242 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:37.157210    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:37.166558 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:37.166525    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:37.513345 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:37.513270    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:37.514211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:37.514192    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:15:38.029625 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:38.029583    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a00028a-af52-4a83-b3d6-176671717fd7" path="/var/lib/kubelet/pods/1a00028a-af52-4a83-b3d6-176671717fd7/volumes"
Nov 03 08:15:44.548209 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:44.548180    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p"]
Nov 03 08:15:44.555136 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:44.554298    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p"
Nov 03 08:15:44.556521 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:44.555539    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p"]
Nov 03 08:15:44.564948 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:44.564913    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989ac148-0bab-4020-939d-e1d087b018a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-228695-2td9p\" (UID: \"989ac148-0bab-4020-939d-e1d087b018a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p"
Nov 03 08:15:44.565079 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:44.565060    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6qps\" (UniqueName: \"kubernetes.io/projected/989ac148-0bab-4020-939d-e1d087b018a5-kube-api-access-w6qps\") pod \"maas-api-revoke-keys-e2e-api-b-228695-2td9p\" (UID: \"989ac148-0bab-4020-939d-e1d087b018a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p"
Nov 03 08:15:44.565134 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:44.565119    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989ac148-0bab-4020-939d-e1d087b018a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-228695-2td9p\" (UID: \"989ac148-0bab-4020-939d-e1d087b018a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p"
Nov 03 08:15:44.567849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:44.567823    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:15:44.567964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:44.567825    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:15:44.573150 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:44.572970    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:15:44.573150 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:44.573116    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:15:44.667625 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:44.667591    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989ac148-0bab-4020-939d-e1d087b018a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-228695-2td9p\" (UID: \"989ac148-0bab-4020-939d-e1d087b018a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p"
Nov 03 08:15:44.667850 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:44.667718    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6qps\" (UniqueName: \"kubernetes.io/projected/989ac148-0bab-4020-939d-e1d087b018a5-kube-api-access-w6qps\") pod \"maas-api-revoke-keys-e2e-api-b-228695-2td9p\" (UID: \"989ac148-0bab-4020-939d-e1d087b018a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p"
Nov 03 08:15:44.667850 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:44.667788    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989ac148-0bab-4020-939d-e1d087b018a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-228695-2td9p\" (UID: \"989ac148-0bab-4020-939d-e1d087b018a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p"
Nov 03 08:15:44.668625 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:44.668554    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989ac148-0bab-4020-939d-e1d087b018a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-228695-2td9p\" (UID: \"989ac148-0bab-4020-939d-e1d087b018a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p"
Nov 03 08:15:44.670237 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:44.670212    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989ac148-0bab-4020-939d-e1d087b018a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-228695-2td9p\" (UID: \"989ac148-0bab-4020-939d-e1d087b018a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p"
Nov 03 08:15:44.675954 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:44.675926    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6qps\" (UniqueName: \"kubernetes.io/projected/989ac148-0bab-4020-939d-e1d087b018a5-kube-api-access-w6qps\") pod \"maas-api-revoke-keys-e2e-api-b-228695-2td9p\" (UID: \"989ac148-0bab-4020-939d-e1d087b018a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p"
Nov 03 08:15:44.875653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:44.875619    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p"
Nov 03 08:15:45.170213 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:45.170190    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p"]
Nov 03 08:15:45.170965 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:15:45.170945    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989ac148_0bab_4020_939d_e1d087b018a5.slice/crio-db48ff9f82bfafde3cd6098589b74bf31d18487f182c872a6bb8becd33fdfa00 WatchSource:0}: Error finding container db48ff9f82bfafde3cd6098589b74bf31d18487f182c872a6bb8becd33fdfa00: Status 404 returned error can't find the container with id db48ff9f82bfafde3cd6098589b74bf31d18487f182c872a6bb8becd33fdfa00
Nov 03 08:15:45.678297 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:45.678264    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p" event={"ID":"989ac148-0bab-4020-939d-e1d087b018a5","Type":"ContainerStarted","Data":"82c16f70c7cb07913f06ba57165da72e1281fe64013d19b173097cafa3e89784"}
Nov 03 08:15:45.678714 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:45.678316    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p" event={"ID":"989ac148-0bab-4020-939d-e1d087b018a5","Type":"ContainerStarted","Data":"db48ff9f82bfafde3cd6098589b74bf31d18487f182c872a6bb8becd33fdfa00"}
Nov 03 08:15:45.698805 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:45.698730    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p" podStartSLOduration=1.698717627 podStartE2EDuration="1.698717627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:45.693332309 +0000 UTC m=+1343.807796240" watchObservedRunningTime="2026-11-03 08:15:45.698717627 +0000 UTC m=+1343.813181557"
Nov 03 08:15:46.179396 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.179367    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"]
Nov 03 08:15:46.184570 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.184538    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.204044 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.204018    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"]
Nov 03 08:15:46.213637 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.213612    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-616e46-openshift-default-dockercfg-mqp6b\""
Nov 03 08:15:46.222146 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.222116    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.222277 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.222202    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.222344 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.222299    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e519992-15af-4d72-ba82-190c90ab0aa8-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.222407 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.222358    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.222407 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.222386    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.222643 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.222622    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.222917 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.222896    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.223008 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.222945    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.223008 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.222977    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpx2h\" (UniqueName: \"kubernetes.io/projected/5e519992-15af-4d72-ba82-190c90ab0aa8-kube-api-access-gpx2h\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.330783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.330233    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.330783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.330302    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpx2h\" (UniqueName: \"kubernetes.io/projected/5e519992-15af-4d72-ba82-190c90ab0aa8-kube-api-access-gpx2h\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.330783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.330381    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.330783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.330480    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.330783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.330556    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e519992-15af-4d72-ba82-190c90ab0aa8-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.330783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.330614    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.330783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.330641    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.331286 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.330813    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.331286 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.331069    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.362789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.331613    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.362789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.332367    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.362789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.332643    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.362789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.333239    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.362789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.338626    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.362789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.339344    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e519992-15af-4d72-ba82-190c90ab0aa8-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.381097 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.381066    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.384776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.382486    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpx2h\" (UniqueName: \"kubernetes.io/projected/5e519992-15af-4d72-ba82-190c90ab0aa8-kube-api-access-gpx2h\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.384776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.383285    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.555087 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.554995    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:46.802937 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.802900    5070 generic.go:356] "Generic (PLEG): container finished" podID="989ac148-0bab-4020-939d-e1d087b018a5" containerID="82c16f70c7cb07913f06ba57165da72e1281fe64013d19b173097cafa3e89784" exitCode=0
Nov 03 08:15:46.803468 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:46.803440    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p" event={"ID":"989ac148-0bab-4020-939d-e1d087b018a5","Type":"ContainerDied","Data":"82c16f70c7cb07913f06ba57165da72e1281fe64013d19b173097cafa3e89784"}
Nov 03 08:15:47.123918 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:47.123882    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"]
Nov 03 08:15:47.124474 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:15:47.124450    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e519992_15af_4d72_ba82_190c90ab0aa8.slice/crio-165f279145bf9a92f6d81d8c549c8fa3cff51c6cd985b46f9b80d8a816000313 WatchSource:0}: Error finding container 165f279145bf9a92f6d81d8c549c8fa3cff51c6cd985b46f9b80d8a816000313: Status 404 returned error can't find the container with id 165f279145bf9a92f6d81d8c549c8fa3cff51c6cd985b46f9b80d8a816000313
Nov 03 08:15:47.128008 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:47.127966    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:15:47.128079 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:47.128053    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:15:47.128119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:47.128095    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:15:47.814307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:47.814262    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm" event={"ID":"5e519992-15af-4d72-ba82-190c90ab0aa8","Type":"ContainerStarted","Data":"409bb03d7b86506848bbbbc07a8c1b45dc59431ec30d50f76ee16014d3da0711"}
Nov 03 08:15:47.814738 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:47.814329    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm" event={"ID":"5e519992-15af-4d72-ba82-190c90ab0aa8","Type":"ContainerStarted","Data":"165f279145bf9a92f6d81d8c549c8fa3cff51c6cd985b46f9b80d8a816000313"}
Nov 03 08:15:47.833011 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:47.832953    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm" podStartSLOduration=1.832939003 podStartE2EDuration="1.832939003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:15:47.830383716 +0000 UTC m=+1345.944847659" watchObservedRunningTime="2026-11-03 08:15:47.832939003 +0000 UTC m=+1345.947402934"
Nov 03 08:15:48.097281 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:48.093403    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p"
Nov 03 08:15:48.204099 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:48.204071    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/989ac148-0bab-4020-939d-e1d087b018a5-kube-api-access-w6qps\" (UniqueName: \"kubernetes.io/projected/989ac148-0bab-4020-939d-e1d087b018a5-kube-api-access-w6qps\") pod \"989ac148-0bab-4020-939d-e1d087b018a5\" (UID: \"989ac148-0bab-4020-939d-e1d087b018a5\") "
Nov 03 08:15:48.204304 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:48.204291    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/989ac148-0bab-4020-939d-e1d087b018a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989ac148-0bab-4020-939d-e1d087b018a5-maas-api-openshift-ca\") pod \"989ac148-0bab-4020-939d-e1d087b018a5\" (UID: \"989ac148-0bab-4020-939d-e1d087b018a5\") "
Nov 03 08:15:48.204351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:48.204325    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/989ac148-0bab-4020-939d-e1d087b018a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989ac148-0bab-4020-939d-e1d087b018a5-maas-api-xks-ca\") pod \"989ac148-0bab-4020-939d-e1d087b018a5\" (UID: \"989ac148-0bab-4020-939d-e1d087b018a5\") "
Nov 03 08:15:48.204659 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:48.204631    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/989ac148-0bab-4020-939d-e1d087b018a5-maas-api-openshift-ca" pod "989ac148-0bab-4020-939d-e1d087b018a5" (UID: "989ac148-0bab-4020-939d-e1d087b018a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:15:48.205003 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:48.204985    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/989ac148-0bab-4020-939d-e1d087b018a5-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:48.206279 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:48.206259    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/989ac148-0bab-4020-939d-e1d087b018a5-kube-api-access-w6qps" pod "989ac148-0bab-4020-939d-e1d087b018a5" (UID: "989ac148-0bab-4020-939d-e1d087b018a5"). InnerVolumeSpecName "kube-api-access-w6qps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:15:48.206410 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:48.206393    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/989ac148-0bab-4020-939d-e1d087b018a5-maas-api-xks-ca" pod "989ac148-0bab-4020-939d-e1d087b018a5" (UID: "989ac148-0bab-4020-939d-e1d087b018a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:15:48.306959 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:48.306930    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/989ac148-0bab-4020-939d-e1d087b018a5-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:48.306959 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:48.306957    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6qps\" (UniqueName: \"kubernetes.io/projected/989ac148-0bab-4020-939d-e1d087b018a5-kube-api-access-w6qps\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:15:48.557452 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:48.557407    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:48.565611 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:48.565582    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:48.824623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:48.824550    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p"
Nov 03 08:15:48.824623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:48.824587    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p" event={"ID":"989ac148-0bab-4020-939d-e1d087b018a5","Type":"ContainerDied","Data":"db48ff9f82bfafde3cd6098589b74bf31d18487f182c872a6bb8becd33fdfa00"}
Nov 03 08:15:48.825148 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:48.824634    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db48ff9f82bfafde3cd6098589b74bf31d18487f182c872a6bb8becd33fdfa00"
Nov 03 08:15:48.825148 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:48.824888    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:48.825795 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:48.825735    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:15:51.857591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:51.857558    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:15:52.443739 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:15:52.440129    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:15:59.274713 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:59.274634    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:15:59.454876 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:59.454643    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:15:59.516833 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:59.508827    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:15:59.533333 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:15:59.533222    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:16:03.826895 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:03.826851    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:03.827574 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:03.826907    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:07.985920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:07.985883    5070 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-n5bqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:07.986446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:07.986090    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg" podUID="63d796bd-8897-4067-b1b4-e39aecd06b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:10.885268 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:10.885233    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"]
Nov 03 08:16:10.886516 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:10.886480    5070 cpu_manager.go:414] "containerMap: removing container" podUID="989ac148-0bab-4020-939d-e1d087b018a5" containerName="revoke-keys"
Nov 03 08:16:10.886516 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:10.886509    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="989ac148-0bab-4020-939d-e1d087b018a5" containerName="revoke-keys"
Nov 03 08:16:10.886841 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:10.886822    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="989ac148-0bab-4020-939d-e1d087b018a5" containerName="revoke-keys"
Nov 03 08:16:10.907503 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:10.907452    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"]
Nov 03 08:16:10.907638 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:10.907599    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.019031 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.018989    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-data\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.019233 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.019121    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-credential-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.019233 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.019167    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-workload-certs\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.019233 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.019196    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-podinfo\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.019233 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.019218    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istiod-ca-cert\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.019491 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.019372    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-workload-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.019556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.019492    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmmq6\" (UniqueName: \"kubernetes.io/projected/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-kube-api-access-kmmq6\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.020114 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.019946    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-envoy\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.020114 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.020071    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-token\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.122187 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.122155    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-envoy\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.122356 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.122221    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-token\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.122356 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.122242    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-data\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.122356 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.122273    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-credential-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.122356 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.122294    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-workload-certs\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.122356 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.122313    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-podinfo\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.122356 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.122338    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istiod-ca-cert\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.122712 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.122480    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-workload-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.122712 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.122524    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmmq6\" (UniqueName: \"kubernetes.io/projected/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-kube-api-access-kmmq6\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.122712 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.122699    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-data\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.122959 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.122836    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-credential-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.122959 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.122862    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-workload-certs\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.123051 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.123030    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-workload-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.123261 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.123242    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istiod-ca-cert\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.145509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.145452    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-envoy\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.145828 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.145809    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-podinfo\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.145902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.145837    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmmq6\" (UniqueName: \"kubernetes.io/projected/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-kube-api-access-kmmq6\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.145942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.145901    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-token\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:11.262785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:11.251088    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:12.503278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:12.502403    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:16:12.503278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:12.502507    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:16:12.503278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:12.502548    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:16:12.526363 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:12.526305    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"]
Nov 03 08:16:13.025941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:13.025908    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j"]
Nov 03 08:16:13.030328 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:13.030307    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04f1d10-k2h4j"]
Nov 03 08:16:13.494429 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:13.494390    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm" event={"ID":"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9","Type":"ContainerStarted","Data":"e4d3100066a6b2b6e4b419aef7f8dd47e9de612dd61cc0a1256e740e23b50e46"}
Nov 03 08:16:13.494627 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:13.494473    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm" event={"ID":"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9","Type":"ContainerStarted","Data":"9f944b07db8142203ef7ea2fe5beaa7e91d6e1edd87b1c8640dfd837a24998e7"}
Nov 03 08:16:13.535595 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:13.535488    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm" podStartSLOduration=3.5354737739999997 podStartE2EDuration="3.535473774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:16:13.532532379 +0000 UTC m=+1371.646996307" watchObservedRunningTime="2026-11-03 08:16:13.535473774 +0000 UTC m=+1371.649937705"
Nov 03 08:16:14.028875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.028832    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11b28097-fb26-499a-a6da-3dac05bee77a" path="/var/lib/kubelet/pods/11b28097-fb26-499a-a6da-3dac05bee77a/volumes"
Nov 03 08:16:14.252551 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.252500    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:14.254025 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.253994    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 03 08:16:14.254178 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.254071    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm" podUID="be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 03 08:16:14.281843 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.277149    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:14.281843 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.277218    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:14.343393 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.343357    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"]
Nov 03 08:16:14.390841 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.390745    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"]
Nov 03 08:16:14.391042 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.390883    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.505593 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.505560    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fhmh\" (UniqueName: \"kubernetes.io/projected/9de4f8ca-60d4-443b-9307-e7c763130938-kube-api-access-2fhmh\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.505805 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.505614    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de4f8ca-60d4-443b-9307-e7c763130938-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.505805 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.505712    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.505805 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.505791    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de4f8ca-60d4-443b-9307-e7c763130938-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.505972 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.505950    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.506143 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.506123    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de4f8ca-60d4-443b-9307-e7c763130938-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.506183 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.506160    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.506303 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.506285    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.506357 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.506343    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.608520 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.608483    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de4f8ca-60d4-443b-9307-e7c763130938-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.609061 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.608533    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.609061 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.608633    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.609061 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.608720    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.609061 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.608855    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fhmh\" (UniqueName: \"kubernetes.io/projected/9de4f8ca-60d4-443b-9307-e7c763130938-kube-api-access-2fhmh\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.609061 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.608942    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de4f8ca-60d4-443b-9307-e7c763130938-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.609061 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.608977    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.609061 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.609028    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de4f8ca-60d4-443b-9307-e7c763130938-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.609516 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.609073    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.609516 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.609099    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.609516 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.609135    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.609516 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.609324    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de4f8ca-60d4-443b-9307-e7c763130938-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.609741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.609531    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.609741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.609667    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.611640 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.611618    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.612165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.612139    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de4f8ca-60d4-443b-9307-e7c763130938-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.617102 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.617075    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de4f8ca-60d4-443b-9307-e7c763130938-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.617185 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.617138    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fhmh\" (UniqueName: \"kubernetes.io/projected/9de4f8ca-60d4-443b-9307-e7c763130938-kube-api-access-2fhmh\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:14.785229 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:14.785187    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:15.151538 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.151461    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"]
Nov 03 08:16:15.187527 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.186250    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.198592 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.198564    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"]
Nov 03 08:16:15.233108 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:16:15.232674    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de4f8ca_60d4_443b_9307_e7c763130938.slice/crio-bca339d40c2c3acdff46691d2831c4a29ad7e0fdf8b396d6b3f920548986b755 WatchSource:0}: Error finding container bca339d40c2c3acdff46691d2831c4a29ad7e0fdf8b396d6b3f920548986b755: Status 404 returned error can't find the container with id bca339d40c2c3acdff46691d2831c4a29ad7e0fdf8b396d6b3f920548986b755
Nov 03 08:16:15.238070 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.236843    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:16:15.238070 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.236929    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:16:15.238070 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.236994    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:16:15.239587 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.239564    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"]
Nov 03 08:16:15.246489 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.246461    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-envoy\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.246632 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.246532    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-workload-certs\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.246632 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.246607    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-data\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.246822 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.246646    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rw5b\" (UniqueName: \"kubernetes.io/projected/870ca6ae-460c-4c9e-9d51-5140a7025bc5-kube-api-access-2rw5b\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.246822 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.246731    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-workload-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.246975 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.246816    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-credential-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.247046 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.247015    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-token\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.247133 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.247114    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istiod-ca-cert\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.247356 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.247325    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-podinfo\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.252924 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.252901    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 03 08:16:15.253019 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.252957    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm" podUID="be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 03 08:16:15.348639 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.348608    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-podinfo\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.348810 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.348686    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-envoy\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.348810 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.348738    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-workload-certs\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.348949 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.348824    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-data\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.348949 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.348868    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rw5b\" (UniqueName: \"kubernetes.io/projected/870ca6ae-460c-4c9e-9d51-5140a7025bc5-kube-api-access-2rw5b\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.349062 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.348960    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-workload-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.349062 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.348996    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-credential-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.349207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.349173    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-token\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.349207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.349204    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istiod-ca-cert\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.349279 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.349261    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-workload-certs\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.349612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.349558    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-credential-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.349612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.349585    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-data\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.349612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.349600    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-workload-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.350165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.350141    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istiod-ca-cert\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.351208 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.351184    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-envoy\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.351597 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.351578    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-podinfo\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.362793 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.356123    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-token\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.362793 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.356262    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rw5b\" (UniqueName: \"kubernetes.io/projected/870ca6ae-460c-4c9e-9d51-5140a7025bc5-kube-api-access-2rw5b\") pod \"e2e-api-b-228695-openshift-default-547df686cf-hq657\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.519149 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.519063    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss" event={"ID":"9de4f8ca-60d4-443b-9307-e7c763130938","Type":"ContainerStarted","Data":"70c1b4eae07e31e0c089d4f896026fc73ce3527a0c3fd1ea6370da4d5460c15d"}
Nov 03 08:16:15.519149 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.519118    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss" event={"ID":"9de4f8ca-60d4-443b-9307-e7c763130938","Type":"ContainerStarted","Data":"bca339d40c2c3acdff46691d2831c4a29ad7e0fdf8b396d6b3f920548986b755"}
Nov 03 08:16:15.549993 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.546377    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss" podStartSLOduration=1.546358831 podStartE2EDuration="1.546358831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:16:15.537717202 +0000 UTC m=+1373.652181134" watchObservedRunningTime="2026-11-03 08:16:15.546358831 +0000 UTC m=+1373.660822764"
Nov 03 08:16:15.590850 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.589969    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:15.800036 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:15.796478    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:16.074903 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:16.049563    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"]
Nov 03 08:16:16.074903 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:16:16.050062    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod870ca6ae_460c_4c9e_9d51_5140a7025bc5.slice/crio-c3c6f2dd2b7e88a5894f4b2a43acec5171e401dd103ea30af7c3892578a491d6 WatchSource:0}: Error finding container c3c6f2dd2b7e88a5894f4b2a43acec5171e401dd103ea30af7c3892578a491d6: Status 404 returned error can't find the container with id c3c6f2dd2b7e88a5894f4b2a43acec5171e401dd103ea30af7c3892578a491d6
Nov 03 08:16:16.074903 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:16.056052    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:16:16.074903 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:16.056138    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:16:16.074903 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:16.056183    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:16:16.252297 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:16.252259    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 03 08:16:16.252466 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:16.252324    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm" podUID="be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 03 08:16:16.534217 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:16.534172    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657" event={"ID":"870ca6ae-460c-4c9e-9d51-5140a7025bc5","Type":"ContainerStarted","Data":"283a6618d7b74aa3c22c7e8daea17b88f1a06ebefb40f4b446c0be1e4beeabc2"}
Nov 03 08:16:16.534418 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:16.534245    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657" event={"ID":"870ca6ae-460c-4c9e-9d51-5140a7025bc5","Type":"ContainerStarted","Data":"c3c6f2dd2b7e88a5894f4b2a43acec5171e401dd103ea30af7c3892578a491d6"}
Nov 03 08:16:16.554426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:16.554342    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657" podStartSLOduration=1.554323406 podStartE2EDuration="1.554323406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:16:16.551097491 +0000 UTC m=+1374.665561421" watchObservedRunningTime="2026-11-03 08:16:16.554323406 +0000 UTC m=+1374.668787339"
Nov 03 08:16:16.591379 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:16.590917    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:16.792819 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:16.792112    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 03 08:16:16.792819 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:16.792189    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss" podUID="9de4f8ca-60d4-443b-9307-e7c763130938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 03 08:16:17.255285 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:17.255251    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 03 08:16:17.255967 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:17.255927    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm" podUID="be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 03 08:16:17.633072 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:17.631942    5070 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-hq657 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 03 08:16:17.633072 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:17.632002    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657" podUID="870ca6ae-460c-4c9e-9d51-5140a7025bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 03 08:16:17.799894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:17.788123    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 03 08:16:17.799894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:17.788196    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss" podUID="9de4f8ca-60d4-443b-9307-e7c763130938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 03 08:16:18.253644 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:18.253190    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 03 08:16:18.253644 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:18.253255    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm" podUID="be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 03 08:16:18.613836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:18.603169    5070 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-hq657 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 03 08:16:18.613836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:18.603228    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657" podUID="870ca6ae-460c-4c9e-9d51-5140a7025bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 03 08:16:18.787568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:18.787127    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 03 08:16:18.787568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:18.787186    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss" podUID="9de4f8ca-60d4-443b-9307-e7c763130938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 03 08:16:18.847833 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:18.847799    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:18.862394 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:18.848155    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:18.944846 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:18.906082    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m"]
Nov 03 08:16:18.985007 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:18.971006    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m"]
Nov 03 08:16:18.985007 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:18.971128    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m"
Nov 03 08:16:19.021545 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:19.001103    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:16:19.021545 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:19.001287    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:16:19.021545 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:19.001354    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:16:19.021545 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:19.001967    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:16:19.086996 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:19.086072    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f04061-9793-4d5f-8189-141d7a74d515-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-950e06-hxf9m\" (UID: \"30f04061-9793-4d5f-8189-141d7a74d515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m"
Nov 03 08:16:19.086996 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:19.086338    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2z2v\" (UniqueName: \"kubernetes.io/projected/30f04061-9793-4d5f-8189-141d7a74d515-kube-api-access-x2z2v\") pod \"maas-api-revoke-keys-e2e-api-a-950e06-hxf9m\" (UID: \"30f04061-9793-4d5f-8189-141d7a74d515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m"
Nov 03 08:16:19.086996 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:19.086459    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f04061-9793-4d5f-8189-141d7a74d515-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-950e06-hxf9m\" (UID: \"30f04061-9793-4d5f-8189-141d7a74d515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m"
Nov 03 08:16:19.225151 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:19.210433    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2z2v\" (UniqueName: \"kubernetes.io/projected/30f04061-9793-4d5f-8189-141d7a74d515-kube-api-access-x2z2v\") pod \"maas-api-revoke-keys-e2e-api-a-950e06-hxf9m\" (UID: \"30f04061-9793-4d5f-8189-141d7a74d515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m"
Nov 03 08:16:19.225151 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:19.210606    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f04061-9793-4d5f-8189-141d7a74d515-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-950e06-hxf9m\" (UID: \"30f04061-9793-4d5f-8189-141d7a74d515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m"
Nov 03 08:16:19.225151 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:19.210785    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f04061-9793-4d5f-8189-141d7a74d515-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-950e06-hxf9m\" (UID: \"30f04061-9793-4d5f-8189-141d7a74d515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m"
Nov 03 08:16:19.225151 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:19.211710    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f04061-9793-4d5f-8189-141d7a74d515-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-950e06-hxf9m\" (UID: \"30f04061-9793-4d5f-8189-141d7a74d515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m"
Nov 03 08:16:19.455532 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:19.254271    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused" start-of-body=
Nov 03 08:16:19.455532 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:19.254334    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm" podUID="be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.197:15021/healthz/ready\": dial tcp 10.132.0.197:15021: connect: connection refused"
Nov 03 08:16:19.688084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:19.653036    5070 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-hq657 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 03 08:16:19.688084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:19.653093    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657" podUID="870ca6ae-460c-4c9e-9d51-5140a7025bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 03 08:16:19.741032 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:19.738455    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2z2v\" (UniqueName: \"kubernetes.io/projected/30f04061-9793-4d5f-8189-141d7a74d515-kube-api-access-x2z2v\") pod \"maas-api-revoke-keys-e2e-api-a-950e06-hxf9m\" (UID: \"30f04061-9793-4d5f-8189-141d7a74d515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m"
Nov 03 08:16:19.767693 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:19.767201    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f04061-9793-4d5f-8189-141d7a74d515-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-950e06-hxf9m\" (UID: \"30f04061-9793-4d5f-8189-141d7a74d515\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m"
Nov 03 08:16:19.813842 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:19.807684    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m"
Nov 03 08:16:19.813842 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:19.810414    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 03 08:16:19.813842 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:19.810463    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss" podUID="9de4f8ca-60d4-443b-9307-e7c763130938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 03 08:16:20.277133 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:20.277048    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:20.412941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:20.284726    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:20.412941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:20.316178    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:16:20.613571 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:20.613012    5070 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-hq657 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 03 08:16:20.613571 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:20.613095    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657" podUID="870ca6ae-460c-4c9e-9d51-5140a7025bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 03 08:16:20.856751 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:20.856318    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 03 08:16:20.856751 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:20.856378    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss" podUID="9de4f8ca-60d4-443b-9307-e7c763130938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 03 08:16:21.032624 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:21.032545    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj"]
Nov 03 08:16:21.640551 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:21.640491    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:16:21.718964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:21.644960    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-974a1057-mz2tj"]
Nov 03 08:16:21.718964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:21.685130    5070 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-hq657 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 03 08:16:21.718964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:21.685249    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657" podUID="870ca6ae-460c-4c9e-9d51-5140a7025bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 03 08:16:21.846829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:21.843983    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Nov 03 08:16:21.846829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:21.844050    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss" podUID="9de4f8ca-60d4-443b-9307-e7c763130938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Nov 03 08:16:22.641060 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:22.641008    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:16:22.985628 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:22.985065    5070 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-n5bqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:22.985628 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:22.985153    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg" podUID="63d796bd-8897-4067-b1b4-e39aecd06b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:23.193332 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.193283    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:23.429117 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.429071    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:23.450715 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.450675    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:23.459814 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.455888    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:23.729611 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.728506    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz" podUID="33f324bd-79b1-4540-a090-7e4a9d92f10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 03 08:16:23.729611 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.729051    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.729727    5070 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-hq657 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.729824    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657" podUID="870ca6ae-460c-4c9e-9d51-5140a7025bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.729882    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j" podUID="67eb2350-8895-4e7d-b1f2-e78e1ff5cf23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.730190    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.730181    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.730311    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.730556    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn" podUID="6960b432-0c5c-4c03-bcc5-dec9ff9466c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.730712    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4" podUID="d50d86f3-08a1-419c-a77b-d7223e0567c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.751960    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.757364    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.683195439s: [/var/lib/containers/storage/overlay/5410767daccbb05e1c4640c0a5b4b6f8915a05c63293401ff4200af219eac3ea/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.757538    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.679982632s: [/var/lib/containers/storage/overlay/5ccad03ee8461333041a6283573923b83a109193a7d883d9612272a0020f1e1b/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.757571    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.680014584s: [/var/lib/containers/storage/overlay/cd609def55b2733ef9a4f1ec2a8d54240b3815b26375e5b2776fcf16a63ac870/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.757602    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.680043851s: [/var/lib/containers/storage/overlay/f9934612dead8415f42ca48a0c55b4b75b82992c6ae6fd3b13012ad235adf7ae/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.757871    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.645649897s: [/var/lib/containers/storage/overlay/db25817ce8da13bbc80345f9dc3dded60bac5f04837b3f3667fdffb152bdfcd5/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.758128    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.645902396s: [/var/lib/containers/storage/overlay/7e4b058ad30d9759aabe88e7b9e1b7556c976ed5edaeef4ff75815c0a6db25f5/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.758363    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.646137585s: [/var/lib/containers/storage/overlay/cc2719fbfc1e579d98cb5ee6c74afe4d10d38eda6ad0548521668719f338c30f/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.758664    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.646436212s: [/var/lib/containers/storage/overlay/a183ff099ac91960dbd85bb3dc5ffdc757a6c36025dcca98834cbde99350d607/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.758722    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.646479067s: [/var/lib/containers/storage/overlay/0be26c8f53132215ed52640745b968751f0e7d0d201203a136e9c20e2530a203/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.758779    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.646534147s: [/var/lib/containers/storage/overlay/4abe9115c689c7fb6e65478d47063788bdf2b5b7724ed1d74bac978658f7e3e9/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.758912    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.645599111s: [/var/lib/containers/storage/overlay/a4270d8fb64b91e5707fa116c0744d078df42426f73a6974da706e38aade4d47/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.758946    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.644569429s: [/var/lib/containers/storage/overlay/122789e593c6abfee289c967980f05aca5f6077c992b85fc1ef098b7799ea843/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.759168    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.644788096s: [/var/lib/containers/storage/overlay/b45d1a0f01c3dc7e7b90f205d9bad2809476ef054ac7700e5a814dda927dbe9a/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.759305    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.644923091s: [/var/lib/containers/storage/overlay/723c41d2de212096c9366ce177fb077292b030dfbf019b43a5dc33e24d6f216a/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.760786    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.64307599s: [/var/lib/containers/storage/overlay/fd3ec678bb22877cd6ee4a17f601af75b84256a8711e412895294c889097939f/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762181    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582244427s: [/var/lib/containers/storage/overlay/24bad55667012151608eeb562ea47ad2c7d475cc1b71d097c50bd0dde6f07d34/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762225    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582288452s: [/var/lib/containers/storage/overlay/4f49bf648fce7facebec5727ab8bda8415d3aca2885852cff8d99409209533ee/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762257    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.58231923s: [/var/lib/containers/storage/overlay/0112aaf46ca8ce7b88f352c9e3745ddf4404561a70be5b846b3326a7d3f16fae/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762292    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582352012s: [/var/lib/containers/storage/overlay/0e3174514928ef3c690edfae096728fa93c3ca5aaf0be3ef40e6bdec7f667fa1/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762324    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582382308s: [/var/lib/containers/storage/overlay/7d8f6da5f22551de9fa878c15047849576ea4be21b607c7e10cf47709eab021b/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762356    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582408473s: [/var/lib/containers/storage/overlay/76ef00c08b8cf6cb2394d74724644696ed87ed9cb311af656dadd570d7a4442b/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762389    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.58243983s: [/var/lib/containers/storage/overlay/613b93a2d450d4c53f70484875698b749079155fbd5919b40362ca45eadac294/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762420    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582448113s: [/var/lib/containers/storage/overlay/7970c3440c952a058d29a6ebb2428bcc381cd283c0ceb98c1069d8b0bfe727bf/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762454    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.58248109s: [/var/lib/containers/storage/overlay/84c38be1b8d80164db94bd20ffcf968b433188c892342cb0944d071735022db3/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762486    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582511209s: [/var/lib/containers/storage/overlay/fb2ebd4fa2f2bebd6a547fd0ddf6caf858dfc356c7a4d3352059958fd126be8c/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762518    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582540559s: [/var/lib/containers/storage/overlay/dc56ebd81e9d853bf7878912bfa56002cea58f2042fb15b8e156518762389d5b/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762552    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582573845s: [/var/lib/containers/storage/overlay/a18a0dc2ef2c163b8a214ded1c8b82df0dd5e01a625c602e8d993ed4eb1b7caa/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762583    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.58255037s: [/var/lib/containers/storage/overlay/0b859f37e0750d157706eda7a629819a6d19faaf4a1b99ac0a4d4523286d9cf6/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762617    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582562699s: [/var/lib/containers/storage/overlay/e8c63d6063b4d1c8136a264ed8aa1b111a517af81b80ff6e50cdacc90bbca5d0/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762649    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582593853s: [/var/lib/containers/storage/overlay/d881434cf1f7f0556ff40852563c79e3dc5ac71604abbd358c14b4101783c94f/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762681    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582622588s: [/var/lib/containers/storage/overlay/aecce4180f140ca9d200ca9efa914f4068c62eae0bf094819acb7a8e324126f5/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762715    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582656092s: [/var/lib/containers/storage/overlay/47ca33b29d9e47bf71ccf609bd487a540f1fb93f8494e741c1818ed17070ed95/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762747    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582687176s: [/var/lib/containers/storage/overlay/0cb1132d74172e45c6d7a07535a54e7c399434bbfc9af7021e4ad6e89aef97fb/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762793    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582731517s: [/var/lib/containers/storage/overlay/e54e1ae75bfb6e0b143a8fa80b885244e370e1d118a414bb6d0e59cc17b4f3d8/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762827    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582764122s: [/var/lib/containers/storage/overlay/b0ec23790299599d0f8c138890d6bd8a3b1af93a374d3e3b1b30647bdeba7aa4/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762861    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582796656s: [/var/lib/containers/storage/overlay/04391fe6aaeed2fdc6fe7af7c7df8417d4b47e786dddd0b3e8244159b006d430/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762894    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582815774s: [/var/lib/containers/storage/overlay/f4ee778e4002f4db62a5b0a488c08db35bcebfe2f1e89ba642162153fbfd04b0/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762927    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582846263s: [/var/lib/containers/storage/overlay/71ecb4e75b50335670f2ed2f6caa1eaaad43c76f28dec0ddbd6c38bf00a1dfa3/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762964    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582879779s: [/var/lib/containers/storage/overlay/94148c766aea61614247824fe217e5a4b32b5af869ebcf615414609024e36b9c/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.762997    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582913078s: [/var/lib/containers/storage/overlay/30f481ea6400378e2b49baaeafa2acac4388e2b0db156f30dd671c75688b6b29/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763033    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582938093s: [/var/lib/containers/storage/overlay/9acc0efff1ed0cf9c30d29065e82d972fad814b564d7bba0b674d95f7bf08cbd/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763067    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582969631s: [/var/lib/containers/storage/overlay/fbf70030274ac48911a53d6285a2ccc3c4d58949611add82c4ce78a7832b60c3/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763064    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.582960457s: [/var/lib/containers/storage/overlay/a108563d20abc6fefab7a7607ab469bf8a32ddce0a2555fc31c412f165c86ba3/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763102    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583001526s: [/var/lib/containers/storage/overlay/25d41ab895a74ef6dcacef4e64496c53e4afbc9d3ba3a568fbeeb346568a344b/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763108    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583007681s: [/var/lib/containers/storage/overlay/84c0329639baa7408152f31bc9418e3d6543df3b74155ee5fedefdf90eb9ebc0/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763136    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583032263s: [/var/lib/containers/storage/overlay/0799995acceb617a6dca9eae208f06d702150aa90ed31d07086ed7f79c5b3b37/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763146    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.689264109s: [/var/lib/containers/storage/overlay/4653dadbe5d893f1e130a8692e3a9aa6a1315423d5472c89495000616655eb34/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763170    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583064419s: [/var/lib/containers/storage/overlay/4f6176579255f444da41dc97541607c9b4d0aace0ba60cdead30ae1dfe2212dd/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763182    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583074415s: [/var/lib/containers/storage/overlay/4c7386c3907de616a376237cb9a325fa5c2d46f2b80a59b33bbe9a1f7939122e/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763205    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583096708s: [/var/lib/containers/storage/overlay/49b9e913d8dcc642ceffd22a9593a1ff19230c5873936cac6d438cee33548c51/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763216    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583105238s: [/var/lib/containers/storage/overlay/3ff4042c560d4e9cf943de5fd09fcae314786c420dd8e8dbad52b46416d61dd8/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763239    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583126516s: [/var/lib/containers/storage/overlay/e2303a7b202988c811b3f015ddb7fd445ac3c3dd392e3c27e59d61bafdbc1862/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763252    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583128334s: [/var/lib/containers/storage/overlay/0285d6781d5953823283a4739c5198c62391dc26dbd602dddae068e59ad5114a/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763274    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583148957s: [/var/lib/containers/storage/overlay/1f8a83d72230ae8c697d05d9013ce453fd2cb3b3dbd3125342e2db9f5175fa87/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763286    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583160394s: [/var/lib/containers/storage/overlay/5c4c83aafc3128c0d4f0a258a384d47abf18824d680f188cb8b41bea9f5e7bb3/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763307    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583179016s: [/var/lib/containers/storage/overlay/0b520ea119426ff8cca1280481e8187ba093b0cf0698898697dfc877ad4373e1/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763321    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583191989s: [/var/lib/containers/storage/overlay/78ed8879d2a1adee650ee39b5d168eb49bad79dd28f2f9ad701b508884f6ab93/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763349    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583217593s: [/var/lib/containers/storage/overlay/8d372ea22a7bb37c7a1a8974ac62df2a3ddcf1a80db45e748518d33a323c6ffa/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763364    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583230872s: [/var/lib/containers/storage/overlay/72e5f7e3557989ecefc997be52713baea65a833d907a5628d20b5a19fd7580e1/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763385    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583250874s: [/var/lib/containers/storage/overlay/c66c2a9cd4f7be9b96bf391a4c10786e023b79570ea8bc3649e2e1d11793c2df/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763399    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583263346s: [/var/lib/containers/storage/overlay/ac3e06c8d4975b6dc346821477e9800fefd560ee55f69a15273f9e3cca05fad3/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763418    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583280501s: [/var/lib/containers/storage/overlay/42728d12d6bd91865c0aa092f18d751cda3250f5de60cea16e957d099be104fe/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763432    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583293611s: [/var/lib/containers/storage/overlay/4937784c7622f7d2034a9423792d1d6eaf1ff6c37902d00d4275c743dae74e98/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763454    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583313623s: [/var/lib/containers/storage/overlay/a44b1a1a063fa4ca7ef76b0f3818738faa15c4dfa24d1964899724eb3c9ae638/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763466    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583317902s: [/var/lib/containers/storage/overlay/b5490c59a865b23c3eafd35b7c56a121801261432e8cb1666307164da6e5d8ac/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763487    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583336659s: [/var/lib/containers/storage/overlay/4b72c678c506d0cfd5678c5e40883ee6813419cdc8abfdaf3f3bd308ef76cc95/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763503    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.558621815s: [/var/lib/containers/storage/overlay/334eca98ac21ee453eb2f45f1481e0784319ee1b996574f162ab895fea05c850/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763522    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.556054345s: [/var/lib/containers/storage/overlay/130f5ea6cf6add1a8e5ed71f3e5fa320ef2a87a33c4c1085e1bedead2ada82ac/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763536    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.534381049s: [/var/lib/containers/storage/overlay/ced64365cfef36fddace701838d35dae3140125f8e31bcd59d88731ec2f45fc1/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763558    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.534393841s: [/var/lib/containers/storage/overlay/5f213c69abf9e1d74ae59798ac94c0b661bb11d455aeb0873b80eded99819027/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763573    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.534406202s: [/var/lib/containers/storage/overlay/65c06c0ed7e6a94671a7a95cb80a6cf3cc4bf3712aa49ae54dfab51d119cb8c5/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763596    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.53442818s: [/var/lib/containers/storage/overlay/819cc8b34166e7bb65c85705a8f1ef8b616ae3d9a74cee9d6b4f8a46dd085081/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763606    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.534435653s: [/var/lib/containers/storage/overlay/878cdc7132265ccfca06ae7006ddd6dfc39981f0d382159ab2ebbac13130bccb/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763630    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.534454947s: [/var/lib/containers/storage/overlay/317ee689d858fa39a984b31a2509e7ce27e76ab07ce37afdcb4c4c2977cd8492/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763711    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.689850714s: [/var/lib/containers/storage/overlay/4f86a579e656d5e185ddbfd67b20f5376e13955071934588aba0113425c1dfd7/diff /var/log/pods/openshift-service-ca_service-ca-5fffc97bdf-cnskv_96cfe445-92c7-4bfe-bafd-8943ce48cee8/service-ca-controller/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763740    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.689584101s: [/var/lib/containers/storage/overlay/8fd03e507be5819b5aaf0c41e902974289d3ca4f44b286054277289ebc9e38c2/diff /var/log/pods/openshift-multus_multus-additional-cni-plugins-h4dgx_19a8bde5-584f-4ff9-8e96-df33c5cda82d/kube-multus-additional-cni-plugins/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763808    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.583702819s: [/var/lib/containers/storage/overlay/8d0c3d80a05cd6ac965bc46d1cf113dcaa3a7082c885c19e8f68a1374157badc/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763831    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.670006661s: [/var/lib/containers/storage/overlay/6be60c2f8f113d364391132db743b8aae23d8470eab057c6e2becc9aa9252e0c/diff /var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal_fc5342d3d37d4243b5710bd6f9020da4/haproxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763863    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.6516435s: [/var/lib/containers/storage/overlay/23554b527a207892578b5ea885e6f37d2887a58e5633b436ce13cf5ac4d96894/diff /var/log/pods/openshift-network-diagnostics_network-check-source-6f78d59f99-fshz5_7e349708-a596-40ca-8724-6c2169cace8c/check-endpoints/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763871    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.651648664s: [/var/lib/containers/storage/overlay/d4a5917a6df8ba521b271159356513b6f0d05f4161f98a363ae47c81779b2f04/diff /var/log/pods/kube-system_konnectivity-agent-9j2f9_676416b5-105e-4650-baed-92baf169a9a6/konnectivity-agent/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763882    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.65165834s: [/var/lib/containers/storage/overlay/41c835025f3c0387a06a91631f53be2240ba6f6033517c3e0d63fa99472914ec/diff /var/log/pods/kube-system_kas-connection-checker-685895c7c9-kr2q8_edf55612-a9f0-4fd5-ab00-ac2271acfb30/connection-checker/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763895    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.651669239s: [/var/lib/containers/storage/overlay/15ea62b9e23e8e6800b4fde708995535c4f5b0c434815e166059c50748d59a14/diff /var/log/pods/openshift-network-operator_iptables-alerter-kj8jw_bd5c9498-f849-4dd6-a9e7-27a0d5db4fde/iptables-alerter/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763903    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.650594107s: [/var/lib/containers/storage/overlay/7d869292b0888f5ff3dc97f48873fcf8844ac28b66cd10ca0a637c137ef5392a/diff /var/log/pods/kube-system_kas-connection-checker-685895c7c9-v2z4k_20b9100c-2539-4227-8f2a-c93660540664/connection-checker/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763915    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.649536921s: [/var/lib/containers/storage/overlay/90736672e988d8c473351c64fa7398c7d3125c35bb8b8f8945b86d8c8c0935e3/diff /var/log/pods/kube-system_global-pull-secret-syncer-ljbh2_dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df/global-pull-secret-syncer/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763924    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.649543816s: [/var/lib/containers/storage/overlay/afad74cf4674fd28c7e50e95d776a42a1943e0521067660b40d71e01308b5ca4/diff /var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-fhz8r_1778099a-ebc0-47ba-98d0-2b0e5726e7d2/migrator/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763945    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.64846338s: [/var/lib/containers/storage/overlay/fd651a03db1307e71e6aefa2b96dcf1c28d4f3aad948fce44a040705befaa7ac/diff /var/log/pods/openshift-multus_multus-msq9p_2ed628f2-4323-47b6-ac03-b8567812a740/kube-multus/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763948    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.649555525s: [/var/lib/containers/storage/overlay/c56b88917cb8fe2794ca4dd78eb8f176d7bf0453a60e59b7cabb0d05af4e4c53/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6d47_40259634-6b57-492e-8376-75ecf7900b06/northd/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763963    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.64741464s: [/var/lib/containers/storage/overlay/0f94dd1caa051ccc6216bcf2c3d95eff551d65e6d1a585077d4f05a4169a44cd/diff /var/log/pods/kube-system_kas-connection-checker-685895c7c9-nwtkn_08817fd9-0879-4b77-a767-5f4c4ecf8d5b/connection-checker/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763970    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.647419298s: [/var/lib/containers/storage/overlay/2a127433bc24e0d353f30e5b66b3786ae51f6cba8daea93211cd2a8ffdcaa1a3/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6d47_40259634-6b57-492e-8376-75ecf7900b06/kube-rbac-proxy-node/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763983    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.647431713s: [/var/lib/containers/storage/overlay/52815f58625b3cd9328c514ed3e22d5adbf2b1b3ace224e6ed7d74c6547f09b3/diff /var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-d7zfg_fc09b315-54d3-44f0-b6ea-8028bd3b212b/migrator/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.764001    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.646275999s: [/var/lib/containers/storage/overlay/f51b448218792043c33a2f891d02fca10fb37373a56c0bd98d3dc4903c5a042a/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6d47_40259634-6b57-492e-8376-75ecf7900b06/ovnkube-controller/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.763999    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.646296035s: [/var/lib/containers/storage/overlay/59dd1bb54e67487a6b2ba1782bd882f1cf54a0ef2cfe77934c724b7114a474c6/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-n28db_5738916f-8202-40ed-823e-90f7c9cc4ec9/csi-liveness-probe/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.764020    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.644158156s: [/var/lib/containers/storage/overlay/e24e16e05d128d72aaa22961d852b0b86236ea69a2da3773b736565d57d6a47a/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6d47_40259634-6b57-492e-8376-75ecf7900b06/kube-rbac-proxy-ovn-metrics/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.764028    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.646299105s: [/var/lib/containers/storage/overlay/13c633ea42f83ad665cbd246a9bd6f20b626886e0cea0fa8f2435ee5b032cdeb/diff /var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-140-123.us-east-2.compute.internal_3012e3a7d17ffd2dd3688b2234e0123b/kube-rbac-proxy-crio/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.764040    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.644176551s: [/var/lib/containers/storage/overlay/a7c51e1967906d238700ebbf87d78655831df113181ac6424b1579f3e8881d20/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6d47_40259634-6b57-492e-8376-75ecf7900b06/nbdb/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.764049    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.644183941s: [/var/lib/containers/storage/overlay/2c20b3f4ee8e065b80058f479c836e19d36087526b1dd5787b1cdfcfcc649894/diff /var/log/pods/openshift-ingress-canary_ingress-canary-6ldkm_78b91dbb-d9e9-4524-8827-d74c8c0625d0/serve-healthcheck-canary/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.764064    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.643098406s: [/var/lib/containers/storage/overlay/0387c481cea65b94ac1a0e848102ca59d4e59b79b34944d9421c53795937f7ca/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6d47_40259634-6b57-492e-8376-75ecf7900b06/ovn-controller/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.764070    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.690197043s: [/var/lib/containers/storage/overlay/c74cfe138fb859d911e61a6eab7b56afcdb393dce824de6cd3ae07d81f3f0576/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-n28db_5738916f-8202-40ed-823e-90f7c9cc4ec9/csi-driver/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.764090    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.430225268s: [/var/lib/containers/storage/overlay/ad73caa994380824ec6422a12275d48c81a8b5b10a40621a6d1a61b7f0574f8d/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-l6bvc_e33edac1-844e-4caf-bb59-d63291659555/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.764610    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.690526097s: [/var/lib/containers/storage/overlay/af4349dc30948b6627a8f67972eab96c9f99a6cc3ad0812e1f033921a2973114/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.764647    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.690558105s: [/var/lib/containers/storage/overlay/01a1ead5e21d7a78ff50feeca98130cb7112bee7be88f4fa47b32bbc685927c8/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.764678    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.690576667s: [/var/lib/containers/storage/overlay/668a7cb07edce4971f3a2cd556acc78022f667405533329ee2e8503a6857460c/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.764810    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.690697259s: [/var/lib/containers/storage/overlay/21a3a95e639a08137a997b44665a13023a1ea0cd58deb83638ec38e51ff3255a/diff /var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-zhh6g_3cb721ff-442b-41b7-8287-220dbfbb41da/volume-data-source-validator/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.764841    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.690725333s: [/var/lib/containers/storage/overlay/739d4efed1f52b22357393aaff69920657173e1d2403793ab3a3da626184961f/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.765022    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.690902955s: [/var/lib/containers/storage/overlay/9d21c08e8a5a4369f8b0612295564e8b90812e048b01ce983ba1622b755f7341/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6d47_40259634-6b57-492e-8376-75ecf7900b06/ovn-acl-logging/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.765054    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.690933301s: [/var/lib/containers/storage/overlay/84d02f8bfec30abf1b103b0eb6f1038e3b414b551b4ee14c107da98a38bed71b/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.765087    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.690962812s: [/var/lib/containers/storage/overlay/17856eae4c2ef17b6938c8c40315aafb327e684031791285726c4e3ee986a4e5/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.765118    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.690974238s: [/var/lib/containers/storage/overlay/75435184d976ba77ddee9082d34d056a46d42f8cd6374277020e17200cc70039/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.765524    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.691369388s: [/var/lib/containers/storage/overlay/685224ce7bd4d8c145b472a52705b9e8027285691a3faae59fb13ee3c00774c7/diff /var/log/pods/openshift-dns_node-resolver-drdft_f142ef3a-aa21-48eb-a403-8d51d843c72d/dns-node-resolver/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.765686    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.688140524s: [/var/lib/containers/storage/overlay/733f174aa5da81d763dd09a8c1191427584a3467f545b47c6b2d407310673f2c/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-n28db_5738916f-8202-40ed-823e-90f7c9cc4ec9/csi-node-driver-registrar/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.766209    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.692070953s: [/var/lib/containers/storage/overlay/82213ffe72ed1a1ad81cfeac6f121470b3250f64bc0bc5e397c24216a4884c94/diff /var/log/pods/openshift-network-diagnostics_network-check-target-c6bms_3a9e7292-b781-4a27-8a48-94f0a8843fc2/network-check-target-container/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.766231    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.692073575s: [/var/lib/containers/storage/overlay/31a26b1dac89111d89e7dac7e69adc68ee1da2a90dd99b42a93a13055c553c78/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.773806    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.699959968s: [/var/lib/containers/storage/overlay/71444a2009cecdd6bcfb4d8e2b311b013056992690f3a7c854ca6d2b26d4ec23/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.774617    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.70050582s: [/var/lib/containers/storage/overlay/8392e0a38c8e88baafdf6d5a100f8d9f48f3dd827a120bd45f86aeabf3e76d63/diff /var/log/pods/openshift-cluster-node-tuning-operator_tuned-fgwf4_79888519-e73a-4536-96e4-ce8e3ec99037/tuned/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.784219    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.710101183s: [/var/lib/containers/storage/overlay/f224ccd20c737d10668e59784d8c05fef8e5e66499779529ee12972b2b663974/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.784990    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.71082937s: [/var/lib/containers/storage/overlay/ea1eb3c251e5767f2c68b94122a247c834ac3d10477c1ccb9eef47c7ff1e5f9c/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x6d47_40259634-6b57-492e-8376-75ecf7900b06/sbdb/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.790078    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.794063    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss" podUID="9de4f8ca-60d4-443b-9307-e7c763130938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:23.812944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.794523    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.720373438s: [/var/lib/containers/storage/overlay/adaa2c44db503a949f1fd6ec4c6bab948cce86159d907faf1e5ad23c7ddadd91/diff /var/log/pods/openshift-image-registry_node-ca-w9c2l_67ff6855-c974-451d-b582-679e7f26e98d/node-ca/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:27.672873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.939601    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="158baf44-a9e1-4dba-ae8b-7a980fbd1443" path="/var/lib/kubelet/pods/158baf44-a9e1-4dba-ae8b-7a980fbd1443/volumes"
Nov 03 08:16:27.672873 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:16:23.940221    5070 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.923s"
Nov 03 08:16:27.672873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.940318    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:27.672873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.940481    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:27.672873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.940532    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:16:27.672873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.940564    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:16:27.672873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.946005    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:27.672873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.946065    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:16:27.672873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.946747    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:27.672873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:23.949500    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:16:27.672873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:24.760538    5070 trace.go:236] Trace[715285489]: "iptables ChainExists" (03-Nov-2026 08:16:22.142) (total time: 2350ms):
Nov 03 08:16:27.672873 ip-10-0-140-123 kubenswrapper[5070]: Trace[715285489]: [2.350616359s] [2.350616359s] END
Nov 03 08:16:27.672873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:24.769921    5070 trace.go:236] Trace[247293643]: "iptables ChainExists" (03-Nov-2026 08:16:22.169) (total time: 2321ms):
Nov 03 08:16:27.672873 ip-10-0-140-123 kubenswrapper[5070]: Trace[247293643]: [2.321898637s] [2.321898637s] END
Nov 03 08:16:27.672873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:25.013628    5070 scope.go:122] "RemoveContainer" containerID="37ed135405642f3729b4edb29218e7deb72017f6c217798026d9a7835be0154f"
Nov 03 08:16:27.672873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:26.267455    5070 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-5qpnb container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.18:8443/readyz\": context deadline exceeded" start-of-body=
Nov 03 08:16:27.672873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:26.267528    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" podUID="d83648de-81dd-4160-8fa9-06d36b38d9de" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/readyz\": context deadline exceeded"
Nov 03 08:16:27.672873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:27.529628    5070 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-5qpnb container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.18:8443/healthz\": context deadline exceeded" start-of-body=
Nov 03 08:16:27.672873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:27.529691    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" podUID="d83648de-81dd-4160-8fa9-06d36b38d9de" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/healthz\": context deadline exceeded"
Nov 03 08:16:29.310772 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:29.310700    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw" podUID="0d26093c-a60a-4cd8-9d43-8ecded62d895" containerName="server" probeResult="failure" output="Get \"https://10.132.0.47:8443/readyz\": context deadline exceeded"
Nov 03 08:16:29.404750 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:29.404699    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:29.406934 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:29.406896    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:29.411107 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:29.411081    5070 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-qmpgn container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:29.588383 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:29.411126    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-qmpgn" podUID="5400462b-2298-41e2-b44b-caa2f3a1e496" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:29.588383 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:16:29.413546    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:16:29.588383 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:29.416577    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz" podUID="33f324bd-79b1-4540-a090-7e4a9d92f10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:29.588383 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:29.564162    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:29.952537 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:29.639432    5070 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-n28db container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:29.952537 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:29.639497    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db" podUID="5738916f-8202-40ed-823e-90f7c9cc4ec9" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:30.486668 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:30.484955    5070 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-j5h2j container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": context deadline exceeded" start-of-body=
Nov 03 08:16:30.486668 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:30.485052    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j" podUID="22591a72-cf89-4406-8e20-82fd6d0f6e6a" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": context deadline exceeded"
Nov 03 08:16:30.486668 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:30.486015    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:16:30.486668 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:30.486543    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:16:30.684958 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:30.579263    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j" podUID="67eb2350-8895-4e7d-b1f2-e78e1ff5cf23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:30.684958 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:30.597847    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz" podUID="33f324bd-79b1-4540-a090-7e4a9d92f10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 03 08:16:30.684958 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:30.630626    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:30.684958 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:30.634071    5070 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-j5h2j container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": context deadline exceeded" start-of-body=
Nov 03 08:16:30.869096 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:30.869058    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw" podUID="0d26093c-a60a-4cd8-9d43-8ecded62d895" containerName="server" probeResult="failure" output="Get \"https://10.132.0.47:8443/healthz\": context deadline exceeded"
Nov 03 08:16:30.869242 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:30.869217    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:30.891821 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:30.891752    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn" podUID="6960b432-0c5c-4c03-bcc5-dec9ff9466c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:31.518457 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:31.518418    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m" podUID="cb3722fc-983b-4b33-9269-8e559475aff5" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 03 08:16:31.771461 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:31.503052    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:31.771461 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:31.519232    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:33.499660 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:33.217853    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded"
Nov 03 08:16:35.784321 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:35.784275    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:35.795265 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:35.795225    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": context deadline exceeded"
Nov 03 08:16:35.871454 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:35.795442    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:16:35.871454 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:35.804446    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9" podUID="917243d5-933a-4f61-9e76-72ec95f6a0dd" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 03 08:16:35.871454 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:35.804856    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:35.911427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:35.911388    5070 patch_prober.go:29] interesting pod/router-default-79698c9d55-m2lbh container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.13:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:36.182470 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:35.911444    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-79698c9d55-m2lbh" podUID="ce890601-127d-4995-9487-635e7aab04f8" containerName="router" probeResult="failure" output="Get \"http://10.132.0.13:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:36.182470 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:35.929987    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j" podUID="22591a72-cf89-4406-8e20-82fd6d0f6e6a" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": context deadline exceeded"
Nov 03 08:16:36.993334 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:36.993288    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz" podUID="33f324bd-79b1-4540-a090-7e4a9d92f10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 03 08:16:45.144110 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:42.094943    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:45.144110 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:44.304039    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j" podUID="67eb2350-8895-4e7d-b1f2-e78e1ff5cf23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:46.982923 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:46.254711    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn" podUID="6960b432-0c5c-4c03-bcc5-dec9ff9466c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 03 08:16:46.982923 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:46.873843    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:47.438557 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:47.438507    5070 patch_prober.go:29] interesting pod/image-registry-7d9549b467-hqhvz container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.132.0.31:5000/healthz\": context deadline exceeded" start-of-body=
Nov 03 08:16:47.438722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:47.438579    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7d9549b467-hqhvz" podUID="5211e023-fa82-4010-939e-89afebb9e6b8" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.31:5000/healthz\": context deadline exceeded"
Nov 03 08:16:47.438722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:47.438649    5070 patch_prober.go:29] interesting pod/dns-default-46l5d container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.132.0.22:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:47.438722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:47.438664    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-46l5d" podUID="279b3dd8-58be-43a1-9994-59fa434409f5" containerName="dns" probeResult="failure" output="Get \"http://10.132.0.22:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:47.692952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:47.438803    5070 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-zmb7g container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:47.692952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:47.438821    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g" podUID="9f09e1af-345e-4489-806a-b3aa7b8cfbd8" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.158:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:47.692952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:47.438834    5070 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-n28db container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:47.692952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:47.438877    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db" podUID="5738916f-8202-40ed-823e-90f7c9cc4ec9" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:47.692952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:47.583517    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:16:47.692952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:47.632139    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:47.692952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:47.635884    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:16:47.692952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:47.635977    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4" podUID="d50d86f3-08a1-419c-a77b-d7223e0567c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:47.978534 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:47.978460    5070 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-5qpnb container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.18:8443/readyz\": context deadline exceeded" start-of-body=
Nov 03 08:16:47.978534 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:47.978512    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" podUID="d83648de-81dd-4160-8fa9-06d36b38d9de" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/readyz\": context deadline exceeded"
Nov 03 08:16:48.277738 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.182465    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:48.277738 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.221067    5070 patch_prober.go:29] interesting pod/network-check-target-c6bms container/network-check-target-container namespace/openshift-network-diagnostics: Readiness probe status=failure output="Get \"http://10.132.0.5:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:48.277738 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.221143    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-network-diagnostics/network-check-target-c6bms" podUID="3a9e7292-b781-4a27-8a48-94f0a8843fc2" containerName="network-check-target-container" probeResult="failure" output="Get \"http://10.132.0.5:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:48.277738 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.221594    5070 patch_prober.go:29] interesting pod/image-registry-7d9549b467-hqhvz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="Get \"https://10.132.0.31:5000/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:48.277738 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.221634    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7d9549b467-hqhvz" podUID="5211e023-fa82-4010-939e-89afebb9e6b8" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.31:5000/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:48.612849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.594122    5070 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-qmpgn container/download-server namespace/openshift-console: Liveness probe status=failure output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:48.612849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.594206    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console/downloads-6dd789c8b7-qmpgn" podUID="5400462b-2298-41e2-b44b-caa2f3a1e496" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:48.612849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.594409    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:48.612849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.594529    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw" podUID="0d26093c-a60a-4cd8-9d43-8ecded62d895" containerName="server" probeResult="failure" output="Get \"https://10.132.0.47:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:48.612849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.594665    5070 patch_prober.go:29] interesting pod/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:48.612849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.594693    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f" podUID="157b7ff4-503c-4ecd-908c-2c078e875322" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:48.612849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.594781    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:48.612849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.594840    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q" podUID="b1cb7af3-82c2-482c-aa2d-510c50735024" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:48.612849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.594912    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.624921    5070 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-j5h2j container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": context deadline exceeded" start-of-body=
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.624997    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j" podUID="22591a72-cf89-4406-8e20-82fd6d0f6e6a" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": context deadline exceeded"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.625120    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw" podUID="0d26093c-a60a-4cd8-9d43-8ecded62d895" containerName="server" probeResult="failure" output="Get \"https://10.132.0.47:8443/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.625292    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.625319    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.627819    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz" podUID="33f324bd-79b1-4540-a090-7e4a9d92f10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.627952    5070 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-5qpnb container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.18:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.627979    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" podUID="d83648de-81dd-4160-8fa9-06d36b38d9de" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.627771    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.628184    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j" podUID="67eb2350-8895-4e7d-b1f2-e78e1ff5cf23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.628257    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.628316    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.640448    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.672883    5070 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-qmpgn container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.672967    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-qmpgn" podUID="5400462b-2298-41e2-b44b-caa2f3a1e496" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.812731    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j" podUID="67eb2350-8895-4e7d-b1f2-e78e1ff5cf23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.979301    5070 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:48.979394    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7" podUID="a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.220173    5070 patch_prober.go:29] interesting pod/router-default-79698c9d55-m2lbh container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.13:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.220245    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-79698c9d55-m2lbh" podUID="ce890601-127d-4995-9487-635e7aab04f8" containerName="router" probeResult="failure" output="Get \"http://10.132.0.13:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.221287    5070 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-j5h2j container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.221300    5070 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-5qpnb container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.18:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.221336    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.221344    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" podUID="d83648de-81dd-4160-8fa9-06d36b38d9de" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.221299    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4" podUID="d50d86f3-08a1-419c-a77b-d7223e0567c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.221344    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j" podUID="22591a72-cf89-4406-8e20-82fd6d0f6e6a" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.221500    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.221507    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.221545    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.221556    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.221553    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn" podUID="6960b432-0c5c-4c03-bcc5-dec9ff9466c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.221670    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.221708    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.221734    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s" podUID="f76ed594-a720-4470-85da-02885a26dd4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.236251    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz" podUID="33f324bd-79b1-4540-a090-7e4a9d92f10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:49.237613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.236259    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb" podUID="17267346-b1b3-475b-9d89-1fda6e36aab5" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.594904    5070 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-qmpgn container/download-server namespace/openshift-console: Liveness probe status=failure output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.594985    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console/downloads-6dd789c8b7-qmpgn" podUID="5400462b-2298-41e2-b44b-caa2f3a1e496" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.594926    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw" podUID="0d26093c-a60a-4cd8-9d43-8ecded62d895" containerName="server" probeResult="failure" output="Get \"https://10.132.0.47:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.595066    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.625420    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw" podUID="0d26093c-a60a-4cd8-9d43-8ecded62d895" containerName="server" probeResult="failure" output="Get \"https://10.132.0.47:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.625431    5070 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-j5h2j container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.625494    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j" podUID="22591a72-cf89-4406-8e20-82fd6d0f6e6a" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.628600    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded"
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.628625    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.628639    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j" podUID="67eb2350-8895-4e7d-b1f2-e78e1ff5cf23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.628602    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz" podUID="33f324bd-79b1-4540-a090-7e4a9d92f10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded"
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.628689    5070 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-5qpnb container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.18:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.628718    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" podUID="d83648de-81dd-4160-8fa9-06d36b38d9de" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.641015    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.653194    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:49.813351    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j" podUID="67eb2350-8895-4e7d-b1f2-e78e1ff5cf23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.224963    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.225093    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn" podUID="6960b432-0c5c-4c03-bcc5-dec9ff9466c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.225139    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.225335    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.225357    5070 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-j5h2j container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": context deadline exceeded" start-of-body=
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.225393    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j" podUID="22591a72-cf89-4406-8e20-82fd6d0f6e6a" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": context deadline exceeded"
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.225431    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.225537    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:50.597351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.225613    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.602622    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb" podUID="17267346-b1b3-475b-9d89-1fda6e36aab5" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.603346    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.603467    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz" podUID="33f324bd-79b1-4540-a090-7e4a9d92f10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded"
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.603547    5070 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-n28db container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.603578    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db" podUID="5738916f-8202-40ed-823e-90f7c9cc4ec9" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.603732    5070 patch_prober.go:29] interesting pod/router-default-79698c9d55-m2lbh container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.13:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.603780    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-79698c9d55-m2lbh" podUID="ce890601-127d-4995-9487-635e7aab04f8" containerName="router" probeResult="failure" output="Get \"http://10.132.0.13:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.620949    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.635396    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j" podUID="67eb2350-8895-4e7d-b1f2-e78e1ff5cf23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.638119    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg" podUID="a2bafc12-e2f3-46ee-8f98-01e46c2c2787" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.69:50052\" within 1s: context deadline exceeded"
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:16:50.638293    5070 logging.go:55] [core] [Channel #1736 SubChannel #1739]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.69:50052", ServerName: "10.132.0.69:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.69:50052: operation was canceled"
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.887866    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.993227    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.554537689s: [/var/lib/containers/storage/overlay/fe4eef60999ce3edd966ed30be2c32d497d4464d01b58d4743b9f725bc6a4309/diff /var/log/pods/openshift-ingress_e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j_acaf418a-affb-4714-a5ce-5673ed5be68e/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.993236    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.597952498s: [/var/lib/containers/storage/overlay/27a82bfc0dd1722c7fcd5b43c4d44029512bc664f8e5f2318846a98c49a87ad3/diff /var/log/pods/openshift-ingress_e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5_d1a356ec-fba8-45e1-87ba-13950f3e1fcb/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.993265    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 15.197680237s: [/var/lib/containers/storage/overlay/599416b3e4f465a27c4afa683f80590441e7d65175fc03d7ef02d75e18b44925/diff /var/log/pods/openshift-ingress_e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt_d4174a67-8cbf-452f-b382-8efd46377534/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.993288    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.361381638s: [/var/lib/containers/storage/overlay/3d7309f3d257b5cc8c43bd460bce4a9fe6f13488ecd043c06a14145bb0723636/diff /var/log/pods/openshift-ingress_e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7_a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.993306    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.597851909s: [/var/lib/containers/storage/overlay/11c576b5822bc88492e6ac707e0a2fcf449468ccd4b747dfcfaea5ec89291740/diff /var/log/pods/opendatahub_model-serving-api-75bd89fdfc-mdwfw_0d26093c-a60a-4cd8-9d43-8ecded62d895/server/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.993321    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.361397496s: [/var/lib/containers/storage/overlay/5caf9c7d34b8ea91f5e3f4ae7d3c92eb16ef7d718cfd86dad4a8f5c435c9ef08/diff /var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-gstjn_6960b432-0c5c-4c03-bcc5-dec9ff9466c7/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.993336    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.776970779s: [/var/lib/containers/storage/overlay/440ce592a39e23378bc5187159333deb96349ad0d272d989b03b915527e15c5d/diff /var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-l99g7_bcbcf547-bdcb-46f9-bd2c-9164868a80ec/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.993351    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.775997218s: [/var/lib/containers/storage/overlay/7c6a67724536acecd567aad1f3e51f578464e062e0abd878776af5b999a06692/diff /var/log/pods/openshift-ingress_e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r_8615eb5b-41a4-4c01-b550-c1d3df743738/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.993367    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.77248034s: [/var/lib/containers/storage/overlay/8113bdb35475ccae9fa7ed4971bdd1037d93cf3ea3d854308bb4cd335922e15a/diff /var/log/pods/openshift-ingress_e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7_17e8f246-f7e4-4dfa-afef-0fb361672461/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.993381    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.772492041s: [/var/lib/containers/storage/overlay/2491086b90bb938739ba7d0b8b3ea7b82b871edf3d5d282310ab0636982a08ff/diff /var/log/pods/openshift-ingress_e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg_c0822706-f191-47c7-8c87-e373872590e3/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.993396    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.772504372s: [/var/lib/containers/storage/overlay/3dbead8875f8a011c0a7c76d014030c90555126a56cdccea912c03a24fa34c0a/diff /var/log/pods/openshift-ingress_e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8_df4cc63e-36c7-4e6e-87ab-64dea6237e42/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.993410    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 15.208760667s: [/var/lib/containers/storage/overlay/aa0599d70c59f34aa9bc080d4e9ce8ffd5bd9c85f3b10d8142ccfceb6d9b404b/diff /var/log/pods/openshift-ingress_router-default-79698c9d55-m2lbh_ce890601-127d-4995-9487-635e7aab04f8/router/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.993424    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.399212517s: [/var/lib/containers/storage/overlay/91f442e9733fe54061e89d1512cf2f6e0f809fcf38a181f9e933d6ceee46c9b7/diff /var/log/pods/openshift-ingress_e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst_11074ce5-9baa-4b3b-ae8b-b30409e5efbb/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.993440    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.380617659s: [/var/lib/containers/storage/overlay/24d886476a570d2ef5385cad091f556cdf59f3c47b4b0006d0a3f087ce53fa99/diff /var/log/pods/openshift-monitoring_node-exporter-4sbrg_60772c10-1acb-489a-89bc-14388a92c001/node-exporter/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.993454    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.749068558s: [/var/lib/containers/storage/overlay/a461be54fe0151e159cfc914dc50e3784203ff2c94a7e5a6da2bfba41aac40f5/diff /var/log/pods/openshift-monitoring_node-exporter-4sbrg_60772c10-1acb-489a-89bc-14388a92c001/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.993468    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 11.101650375s: [/var/lib/containers/storage/overlay/945e05be5de5762cc7bf1999f5be3b052b7ea179b36cc531f10255af2535fa36/diff /var/log/pods/openshift-insights_insights-runtime-extractor-wd497_6731c200-9eda-44f2-aa13-2b4a8fbda85e/extractor/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.993481    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 9.357587246s: [/var/lib/containers/storage/overlay/a8e1dc1ac0bcde3739b6ea2ddb65b897a9c5a17f430b6ca1e77886c7377e3b98/diff /var/log/pods/openshift-insights_insights-runtime-extractor-wd497_6731c200-9eda-44f2-aa13-2b4a8fbda85e/exporter/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.993495    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 9.357568993s: [/var/lib/containers/storage/overlay/eab74faa0a7187fce9bc568b3cd1f91be8f049cd048ad02b8aec21c0a68508b5/diff /var/log/pods/openshift-console_console-58b4bd6669-gzcxn_873e56e5-1f64-4eec-a578-bf4ddd072869/console/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.994071    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.75891225s: [/var/lib/containers/storage/overlay/9e19e84ef1567b3627f40b270cdccdbdafce2ee7e1fccad161737aca1a89c039/diff /var/log/pods/openshift-ingress_e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm_5e519992-15af-4d72-ba82-190c90ab0aa8/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.994131    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.598771261s: [/var/lib/containers/storage/overlay/4e0c89f34b6713e505cc744fbaed7c406b793e4563b93de172c24b9759e681b4/diff /var/log/pods/cert-manager_cert-manager-597b95cb84-tfqr9_61277e6f-ccc4-4d25-a7ca-3dae456b0cdf/cert-manager-controller/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.994138    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.176106536s: [/var/lib/containers/storage/overlay/1410c9924e51643f2dd6faa772f6e8ee3d8ac64c4c015b0ed88ee971cf00b18c/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.994146    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 13.895244989s: [/var/lib/containers/storage/overlay/cc52d2dc199fe6b8629a3d87e1893b9647234ae811f752d72a45bd1eb05cee87/diff /var/log/pods/openshift-ingress_e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv_e724a396-646c-42bb-9b59-d7ff618e07f8/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.994153    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.998747078s: [/var/lib/containers/storage/overlay/3621ddc002dc0fd08261347f2bb4585fa9be365f4209b084c9d8cb83bbd4c070/diff /var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-zmb7g_9f09e1af-345e-4489-806a-b3aa7b8cfbd8/discovery/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.994447    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.555744975s: [/var/lib/containers/storage/overlay/c8526fc1c6a0dbae17698cc373593852b93456af322c00261645696f58c8ef26/diff /var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/3.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.995049    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.359725379s: [/var/lib/containers/storage/overlay/870f1bd0487d8c267c85e673eee23cfd3aaf175776c068ffc663fd128adf5adc/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.995135    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 4.662019059s: [/var/lib/containers/storage/overlay/5bf3e442337de33ac0df8cd46185651e8958dfb6c8fe6f9902e60080fd3ce0a2/diff /var/log/pods/openshift-ingress_e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs_2abf1628-8c68-450c-a458-eefa78b239b3/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:50.996187    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.364321247s: [/var/lib/containers/storage/overlay/828379029f5b9ac28b7150eccdad905a6641dc774a6111360ddaa8c420ed7a78/diff /var/log/pods/e2e-models-e2e-worker-w2-6bc553_distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l_8ae55a46-f433-42f5-b29b-dc0c51b87de4/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.008000    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.570959702s: [/var/lib/containers/storage/overlay/2ed219856e4c5de5f3cc12cae1e33fc02c3a1f58f6739e3f9c4d62ebfe9be814/diff /var/log/pods/kuadrant-system_authorino-6995574766-s7dx9_b148fccf-b3ae-4b52-9df1-21e85f7e3fa3/authorino/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.015344    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.861831177s: [/var/lib/containers/storage/overlay/4e620c8ec12bb277f0079dbf30f30fe4548dc918c5d7a83467c8d4fd18d677a3/diff /var/log/pods/e2e-models-e2e-worker-w6-bd085d_distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2_d172e36b-89d5-43d3-9ad8-9cc9142d12d0/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.015367    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.861861381s: [/var/lib/containers/storage/overlay/2ced64ed57054f08d4a142bf5f20cc11e7d39335a6eb32b8524eef65fb32334c/diff /var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-j5h2j_22591a72-cf89-4406-8e20-82fd6d0f6e6a/prometheus-operator-admission-webhook/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.022609    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 11.064509137s: [/var/lib/containers/storage/overlay/8ffa29993e868f45de5f73246db036395ad06feb872467b8a924f0a7de86834f/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.023974    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.270158076s: [/var/lib/containers/storage/overlay/ed2a8e56d70141c04a258de06dc39ddd4446d21cee63ecae47ce8c0994e0033d/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.024191    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.269870249s: [/var/lib/containers/storage/overlay/a76dda0bb7c0cd84222858fe3885eeb2af80a77866587f24266528e4fe4aa285/diff /var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-l99g7_bcbcf547-bdcb-46f9-bd2c-9164868a80ec/prometheus-operator/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.042541    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn" podUID="063e3656-6675-4336-a3c1-cf96b568be28" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.66:50052\" within 1s: context deadline exceeded"
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.042568    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb" podUID="d449195e-0ed5-4c8e-ac1b-3242198699f7" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.67:50052\" within 1s: context deadline exceeded"
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.042781    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 7.350078641s: [/var/lib/containers/storage/overlay/20277c4f60367071f221575bf87e718678147f8ca79695e10332addebcdf69dc/diff /var/log/pods/openshift-ingress_e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f_157b7ff4-503c-4ecd-908c-2c078e875322/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:16:51.043200    5070 logging.go:55] [core] [Channel #1737 SubChannel #1744]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.67:50052", ServerName: "10.132.0.67:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.67:50052: operation was canceled"
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.043317    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd" podUID="cb961482-e401-47f5-8aff-f604b364d631" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.64:50052\" within 1s: context deadline exceeded"
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:16:51.043551    5070 logging.go:55] [core] [Channel #1735 SubChannel #1743]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.66:50052", ServerName: "10.132.0.66:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.66:50052: operation was canceled"
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:16:51.044899    5070 logging.go:55] [core] [Channel #1738 SubChannel #1745]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.64:50052", ServerName: "10.132.0.64:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.64:50052: operation was canceled"
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.051914    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44" podUID="56258879-0d2c-457c-a0d2-0e7c378bed50" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 03 08:16:51.099464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.054776    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 15.078538992s: [/var/lib/containers/storage/overlay/c85b76859f3acd0f6bde04ab71005e48ee9c8a438cdfae7c234384ad30a7e9cf/diff /var/log/pods/openshift-insights_insights-runtime-extractor-wd497_6731c200-9eda-44f2-aa13-2b4a8fbda85e/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.194969    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.359609314s: [/var/lib/containers/storage/overlay/5c91ed187ffda639c2fda5da3b7a79abceef7b07d0b3900b4998424a6a61424b/diff /var/log/pods/openshift-ingress_kube-auth-proxy-6cf769c4c7-l2kwc_0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f/kube-auth-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.201296    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.570145173s: [/var/lib/containers/storage/overlay/72e07a84cebb17f3191157a257c11d83a11dd4b3936e0c76a7fd6beb42a717ae/diff /var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4_d50d86f3-08a1-419c-a77b-d7223e0567c7/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.201301    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.04791867s: [/var/lib/containers/storage/overlay/65bac48e2b56c51e4a9fa48174bbddd0625111c4f21a6b02be79035843722505/diff /var/log/pods/openshift-image-registry_image-registry-7d9549b467-hqhvz_5211e023-fa82-4010-939e-89afebb9e6b8/registry/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.201493    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 6.048309757s: [/var/lib/containers/storage/overlay/65c394f2cce2857a2c427a22b155a9a0c5c2a476ca36506ff3bd88a735c7959d/diff /var/log/pods/e2e-models-e2e-worker-w5-ffe7a8_premium-w5-ffe7a8-kserve-79db785d58-2skmc_381033e1-f38c-48ba-9bf2-45ba216df6fb/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.201510    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 14.838278409s: [/var/lib/containers/storage/overlay/c766e873a9bf56f1c962de703128d0b9ce3f240715b90127e7c54b50f6d4345c/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.208528    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.132885189s: [/var/lib/containers/storage/overlay/f99095309d506a390e91e4692df6d00af7bfc4caf64ee435761af1ec3381fd67/diff ]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.211192    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 5.457369669s: [/var/lib/containers/storage/overlay/a9986a9cc72916bcb0c269b79ad799fdfc30eb297127a11422fdeea36f1d6d1b/diff /var/log/pods/e2e-models-e2e-worker-w1-19bb0a_distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5_b6603beb-ef9c-4cd4-96b2-286c95a37353/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.217551    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9" podUID="917243d5-933a-4f61-9e76-72ec95f6a0dd" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.65:50052\" within 1s: context deadline exceeded"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.217686    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m" podUID="cb3722fc-983b-4b33-9269-8e559475aff5" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.68:50052\" within 1s: context deadline exceeded"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.229747    5070 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-qmpgn container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.229802    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-qmpgn" podUID="5400462b-2298-41e2-b44b-caa2f3a1e496" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.29:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.241706    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn" podUID="6960b432-0c5c-4c03-bcc5-dec9ff9466c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.241993    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.246183    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.265325    5070 patch_prober.go:29] interesting pod/dns-default-46l5d container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.132.0.22:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.265449    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-46l5d" podUID="279b3dd8-58be-43a1-9994-59fa434409f5" containerName="dns" probeResult="failure" output="Get \"http://10.132.0.22:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.266155    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.267880    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4" podUID="d50d86f3-08a1-419c-a77b-d7223e0567c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.278888    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.286810    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.286844    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss" podUID="9de4f8ca-60d4-443b-9307-e7c763130938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.288818    5070 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-n28db container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.288854    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db" podUID="5738916f-8202-40ed-823e-90f7c9cc4ec9" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.308955    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.384688    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 7.692001974s: [/var/lib/containers/storage/overlay/17ba97af1a026d580a92d27dbf5162ba699afdc642e8c8e354f71e6dae71428e/diff /var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-6kbrb_17267346-b1b3-475b-9d89-1fda6e36aab5/manager/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.570537    5070 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 11.936329645s: [/var/lib/containers/storage/overlay/5f134ad636542892bb60d054795814d11defd3249c47a6cc5975d73dcbbfe984/diff /var/log/pods/openshift-ingress_e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb_d915ac09-a22a-4e5d-943a-97cdeb2d9c24/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.613016    5070 patch_prober.go:29] interesting pod/console-58b4bd6669-gzcxn container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.132.0.36:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.613073    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-58b4bd6669-gzcxn" podUID="873e56e5-1f64-4eec-a578-bf4ddd072869" containerName="console" probeResult="failure" output="Get \"https://10.132.0.36:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.636334    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j" podUID="67eb2350-8895-4e7d-b1f2-e78e1ff5cf23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.645899    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.653921    5070 patch_prober.go:29] interesting pod/router-default-79698c9d55-m2lbh container/router namespace/openshift-ingress: Liveness probe status=failure output="Get \"http://10.132.0.13:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.653967    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/router-default-79698c9d55-m2lbh" podUID="ce890601-127d-4995-9487-635e7aab04f8" containerName="router" probeResult="failure" output="Get \"http://10.132.0.13:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.663868    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65" podUID="9ecd3f55-84e0-4321-8d19-900388b6a91f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.40:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.671854    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb" podUID="17267346-b1b3-475b-9d89-1fda6e36aab5" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.677481    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg" podUID="a2bafc12-e2f3-46ee-8f98-01e46c2c2787" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.678882    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65" podUID="9ecd3f55-84e0-4321-8d19-900388b6a91f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.40:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.681853    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:51.681893    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:52.050866    5070 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-n5bqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:52.050908    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg" podUID="63d796bd-8897-4067-b1b4-e39aecd06b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:52.092864    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44" podUID="56258879-0d2c-457c-a0d2-0e7c378bed50" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:52.094840    5070 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-hq657 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:52.454049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:52.094875    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657" podUID="870ca6ae-460c-4c9e-9d51-5140a7025bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:52.637523    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j" podUID="67eb2350-8895-4e7d-b1f2-e78e1ff5cf23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:52.655877    5070 patch_prober.go:29] interesting pod/console-58b4bd6669-gzcxn container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.132.0.36:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:52.655928    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-58b4bd6669-gzcxn" podUID="873e56e5-1f64-4eec-a578-bf4ddd072869" containerName="console" probeResult="failure" output="Get \"https://10.132.0.36:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:52.694869    5070 patch_prober.go:29] interesting pod/router-default-79698c9d55-m2lbh container/router namespace/openshift-ingress: Liveness probe status=failure output="Get \"http://10.132.0.13:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:52.694909    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/router-default-79698c9d55-m2lbh" podUID="ce890601-127d-4995-9487-635e7aab04f8" containerName="router" probeResult="failure" output="Get \"http://10.132.0.13:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:52.704856    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65" podUID="9ecd3f55-84e0-4321-8d19-900388b6a91f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.40:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:52.713887    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb" podUID="17267346-b1b3-475b-9d89-1fda6e36aab5" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:52.723836    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:52.723871    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:52.954856    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/limitador-limitador-7c9b87d574-qp9pc" podUID="7d888577-bc72-4581-a858-b43bd5d7290c" containerName="limitador" probeResult="failure" output="Get \"http://10.132.0.45:8080/status\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:52.958464    5070 trace.go:236] Trace[162880691]: "Calculate volume metrics of ca-trust-extracted for pod openshift-image-registry/image-registry-7d9549b467-hqhvz" (03-Nov-2026 08:16:51.640) (total time: 1318ms):
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: Trace[162880691]: [1.318188657s] [1.318188657s] END
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:53.072887    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/limitador-limitador-7c9b87d574-qp9pc" podUID="7d888577-bc72-4581-a858-b43bd5d7290c" containerName="limitador" probeResult="failure" output="Get \"http://10.132.0.45:8080/status\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:53.091855    5070 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-n5bqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:53.091896    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg" podUID="63d796bd-8897-4067-b1b4-e39aecd06b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:53.221751    5070 patch_prober.go:29] interesting pod/image-registry-7d9549b467-hqhvz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="Get \"https://10.132.0.31:5000/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:53.221822    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7d9549b467-hqhvz" podUID="5211e023-fa82-4010-939e-89afebb9e6b8" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.31:5000/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:53.638521    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j" podUID="67eb2350-8895-4e7d-b1f2-e78e1ff5cf23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:53.644875    5070 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-n28db container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:53.644907    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db" podUID="5738916f-8202-40ed-823e-90f7c9cc4ec9" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:53.656994    5070 patch_prober.go:29] interesting pod/console-58b4bd6669-gzcxn container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.132.0.36:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:53.657031    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-58b4bd6669-gzcxn" podUID="873e56e5-1f64-4eec-a578-bf4ddd072869" containerName="console" probeResult="failure" output="Get \"https://10.132.0.36:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:54.306858    5070 patch_prober.go:29] interesting pod/dns-default-46l5d container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.132.0.22:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:54.306920    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-46l5d" podUID="279b3dd8-58be-43a1-9994-59fa434409f5" containerName="dns" probeResult="failure" output="Get \"http://10.132.0.22:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:54.329853    5070 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-n28db container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:54.329900    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db" podUID="5738916f-8202-40ed-823e-90f7c9cc4ec9" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:54.679909    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j" podUID="67eb2350-8895-4e7d-b1f2-e78e1ff5cf23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:54.947603    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:54.947668    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss" podUID="9de4f8ca-60d4-443b-9307-e7c763130938" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:54.983937    5070 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-hq657 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:54.983986    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657" podUID="870ca6ae-460c-4c9e-9d51-5140a7025bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:56.062976    5070 patch_prober.go:29] interesting pod/image-registry-7d9549b467-hqhvz container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.132.0.31:5000/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:56.063041    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7d9549b467-hqhvz" podUID="5211e023-fa82-4010-939e-89afebb9e6b8" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.31:5000/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:56.179860    5070 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-5qpnb container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.18:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:56.179944    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" podUID="d83648de-81dd-4160-8fa9-06d36b38d9de" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:56.732862    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65" podUID="9ecd3f55-84e0-4321-8d19-900388b6a91f" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.40:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:57.396880    5070 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-n28db container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:57.396933    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db" podUID="5738916f-8202-40ed-823e-90f7c9cc4ec9" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:57.708852    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb" podUID="17267346-b1b3-475b-9d89-1fda6e36aab5" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:57.984099    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw" podUID="0d26093c-a60a-4cd8-9d43-8ecded62d895" containerName="server" probeResult="failure" output="Get \"https://10.132.0.47:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:16:58.427652    5070 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="29.691s"
Nov 03 08:16:58.821112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:58.427704    5070 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw"
Nov 03 08:17:03.223236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:59.450345    5070 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="console-operator" containerStatusID={"Type":"cri-o","ID":"41db54c2e9197838faec3ccd184adac0bf784c452fd80ef24684e43bbab81f7a"} pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" containerMessage="Container console-operator failed liveness probe, will be restarted"
Nov 03 08:17:03.223236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:59.450394    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" podUID="d83648de-81dd-4160-8fa9-06d36b38d9de" containerName="console-operator" containerID="cri-o://41db54c2e9197838faec3ccd184adac0bf784c452fd80ef24684e43bbab81f7a" gracePeriod=30
Nov 03 08:17:03.223236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:59.844670    5070 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="server" containerStatusID={"Type":"cri-o","ID":"bfd14b3b1a48099d8a2943d58cb72793f8e8fd272d5702cec8dc340d02fd1f29"} pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw" containerMessage="Container server failed liveness probe, will be restarted"
Nov 03 08:17:03.223236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:16:59.844720    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw" podUID="0d26093c-a60a-4cd8-9d43-8ecded62d895" containerName="server" containerID="cri-o://bfd14b3b1a48099d8a2943d58cb72793f8e8fd272d5702cec8dc340d02fd1f29" gracePeriod=30
Nov 03 08:17:03.223236 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:16:59.851734    5070 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.424s"
Nov 03 08:17:03.223236 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:16:59.851779    5070 kubelet.go:2545] "Skipping pod synchronization" err="container runtime is down"
Nov 03 08:17:03.223236 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:17:00.018634    5070 kubelet.go:2545] "Skipping pod synchronization" err="container runtime is down"
Nov 03 08:17:03.223236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:00.149840    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:17:03.223236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:00.149884    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:03.223236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:01.303193    5070 trace.go:236] Trace[725775037]: "Calculate volume metrics of kube-api-access-2kk6l for pod openshift-kube-storage-version-migrator/migrator-66df5d496b-fhz8r" (03-Nov-2026 08:16:59.871) (total time: 1431ms):
Nov 03 08:17:03.223236 ip-10-0-140-123 kubenswrapper[5070]: Trace[725775037]: [1.431293268s] [1.431293268s] END
Nov 03 08:17:03.223236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:01.359551    5070 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-5qpnb container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.18:8443/readyz\": read tcp 10.132.0.2:60224->10.132.0.18:8443: read: connection reset by peer" start-of-body=
Nov 03 08:17:03.223236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:01.359611    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" podUID="d83648de-81dd-4160-8fa9-06d36b38d9de" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/readyz\": read tcp 10.132.0.2:60224->10.132.0.18:8443: read: connection reset by peer"
Nov 03 08:17:03.223236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:03.001836    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:17:03.223236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:03.002302    5070 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-j5h2j container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": context deadline exceeded" start-of-body=
Nov 03 08:17:03.223236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:03.002366    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j" podUID="22591a72-cf89-4406-8e20-82fd6d0f6e6a" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": context deadline exceeded"
Nov 03 08:17:03.223236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:03.002431    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw" podUID="0d26093c-a60a-4cd8-9d43-8ecded62d895" containerName="server" probeResult="failure" output="Get \"https://10.132.0.47:8443/readyz\": context deadline exceeded"
Nov 03 08:17:03.223236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:03.019114    5070 patch_prober.go:29] interesting pod/image-registry-7d9549b467-hqhvz container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.132.0.31:5000/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:17:03.223236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:03.019147    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7d9549b467-hqhvz" podUID="5211e023-fa82-4010-939e-89afebb9e6b8" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.31:5000/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:03.223236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:03.027450    5070 patch_prober.go:29] interesting pod/image-registry-7d9549b467-hqhvz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="Get \"https://10.132.0.31:5000/healthz\": context deadline exceeded" start-of-body=
Nov 03 08:17:03.223236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:03.027481    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7d9549b467-hqhvz" podUID="5211e023-fa82-4010-939e-89afebb9e6b8" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.31:5000/healthz\": context deadline exceeded"
Nov 03 08:17:05.649041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:03.760670    5070 patch_prober.go:29] interesting pod/router-default-79698c9d55-m2lbh container/router namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.13:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:17:05.649041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:03.760730    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/router-default-79698c9d55-m2lbh" podUID="ce890601-127d-4995-9487-635e7aab04f8" containerName="router" probeResult="failure" output="Get \"http://10.132.0.13:1936/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:05.649041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:03.900393    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz" podUID="33f324bd-79b1-4540-a090-7e4a9d92f10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:05.649041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:03.901212    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:05.649041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:03.901303    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:05.649041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:04.184155    5070 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j"
Nov 03 08:17:05.649041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:04.463773    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:17:05.649041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:04.463820    5070 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 08:17:05.649041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:04.463858    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"]
Nov 03 08:17:05.649041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:04.495532    5070 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="prometheus-operator-admission-webhook" containerStatusID={"Type":"cri-o","ID":"d9fea37b92890625680a72e06e7f19262c344e2ce9b5941ca192b5f45e22ca33"} pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j" containerMessage="Container prometheus-operator-admission-webhook failed liveness probe, will be restarted"
Nov 03 08:17:05.649041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:04.495837    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j" podUID="22591a72-cf89-4406-8e20-82fd6d0f6e6a" containerName="prometheus-operator-admission-webhook" containerID="cri-o://d9fea37b92890625680a72e06e7f19262c344e2ce9b5941ca192b5f45e22ca33" gracePeriod=30
Nov 03 08:17:05.649041 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:17:04.751382    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:17:05.649041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:04.783370    5070 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-j5h2j container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": EOF" start-of-body=
Nov 03 08:17:05.649041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:04.783426    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j" podUID="22591a72-cf89-4406-8e20-82fd6d0f6e6a" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": EOF"
Nov 03 08:17:05.649041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:05.286208    5070 scope.go:122] "RemoveContainer" containerID="3c2ad242be5517969b3f9c1810cd708fe136eeb3bc6b6ccc69e8af73d16d59e6"
Nov 03 08:17:06.094678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:05.907616    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/3.log"
Nov 03 08:17:06.094678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:05.964820    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/2.log"
Nov 03 08:17:06.094678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:05.964897    5070 generic.go:356] "Generic (PLEG): container finished" podID="d83648de-81dd-4160-8fa9-06d36b38d9de" containerID="41db54c2e9197838faec3ccd184adac0bf784c452fd80ef24684e43bbab81f7a" exitCode=1
Nov 03 08:17:06.094678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:05.974645    5070 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 08:17:06.094678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:05.974733    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:17:06.094678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:05.974795    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:17:06.094678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:05.974901    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"]
Nov 03 08:17:06.094678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:05.974945    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"]
Nov 03 08:17:06.094678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:05.982048    5070 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"a0f5d44418a8ed4fdedf0b5f3452626063bda534bdd4c631195fca4a944673aa"} pod="openshift-image-registry/image-registry-7d9549b467-hqhvz" containerMessage="Container registry failed liveness probe, will be restarted"
Nov 03 08:17:06.094678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.026826    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.094678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.054084    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-workload-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.094678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.054140    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-istio-data\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.094678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.054263    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/198146ed-665b-4801-8396-c980ee6ff4d4-istio-podinfo\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.094678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.054349    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-istio-envoy\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.094678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.054447    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-workload-certs\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.094678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.054479    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-credential-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.094678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.054590    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/198146ed-665b-4801-8396-c980ee6ff4d4-istiod-ca-cert\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.094678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.054972    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69lxp\" (UniqueName: \"kubernetes.io/projected/198146ed-665b-4801-8396-c980ee6ff4d4-kube-api-access-69lxp\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.094678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.055008    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/198146ed-665b-4801-8396-c980ee6ff4d4-istio-token\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.156573    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-workload-certs\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.156605    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-credential-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.156669    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/198146ed-665b-4801-8396-c980ee6ff4d4-istiod-ca-cert\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.156842    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69lxp\" (UniqueName: \"kubernetes.io/projected/198146ed-665b-4801-8396-c980ee6ff4d4-kube-api-access-69lxp\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.156870    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/198146ed-665b-4801-8396-c980ee6ff4d4-istio-token\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.156922    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-workload-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.156957    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-istio-data\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.157040    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/198146ed-665b-4801-8396-c980ee6ff4d4-istio-podinfo\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.157114    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-istio-envoy\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.254061    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.254097    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"]
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.265969    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.268613    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-credential-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.297070    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-workload-socket\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.336783    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-workload-certs\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.369146    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.369395    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.369512    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.369747    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.369835    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wgkx\" (UniqueName: \"kubernetes.io/projected/212c33bc-85a6-4ff5-a96a-eda0064fab63-kube-api-access-2wgkx\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.369865    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212c33bc-85a6-4ff5-a96a-eda0064fab63-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.369901    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.370005    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:06.389253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.370058    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.389608    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-istio-data\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.390978    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/198146ed-665b-4801-8396-c980ee6ff4d4-istiod-ca-cert\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.448507    5070 scope.go:122] "RemoveContainer" containerID="63717ea761b062bdd9ba53c28ba1e10cd9bd50a0cfeb7b832ebf629d6d5ebbe5"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.471283    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.471313    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wgkx\" (UniqueName: \"kubernetes.io/projected/212c33bc-85a6-4ff5-a96a-eda0064fab63-kube-api-access-2wgkx\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.471338    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212c33bc-85a6-4ff5-a96a-eda0064fab63-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.471363    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.471419    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.471526    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.471675    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.471856    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.471991    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.663913    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.778422    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b95ll\" (UniqueName: \"kubernetes.io/projected/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-kube-api-access-b95ll\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.778590    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-workload-certs\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.778791    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-data\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.778902    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-envoy\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.778963    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istiod-ca-cert\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.779061    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-token\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.779163    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-podinfo\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.779247    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-workload-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:06.793933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.779504    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-credential-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:06.955091 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.881339    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-workload-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:06.955091 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.881596    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-credential-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:06.955091 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.881715    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b95ll\" (UniqueName: \"kubernetes.io/projected/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-kube-api-access-b95ll\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:06.955091 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.881833    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-workload-certs\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:06.955091 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.881944    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-data\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:06.955091 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.882015    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-envoy\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:06.955091 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.882040    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istiod-ca-cert\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:06.955091 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.882108    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-token\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:06.955091 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.882192    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-podinfo\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.998373    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"]
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.998433    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"]
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.998464    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" event={"ID":"d83648de-81dd-4160-8fa9-06d36b38d9de","Type":"ContainerDied","Data":"41db54c2e9197838faec3ccd184adac0bf784c452fd80ef24684e43bbab81f7a"}
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.998551    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.998594    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.998615    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.998681    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:06.998723    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.005785    5070 scope.go:122] "RemoveContainer" containerID="a10b2a1a54d542458744e9c86460bf61c3c55fcc11db69ea74a2a6b9b035fd86"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.012492    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.015005    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.015837    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.018102    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.018644    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.019201    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.019612    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.021028    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.021110    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.021162    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-qmpgn"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.021223    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.021258    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.021276    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.021313    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.021330    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.021358    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.021389    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.021427    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.023865    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb" podUID="17267346-b1b3-475b-9d89-1fda6e36aab5" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": dial tcp 10.132.0.42:8081: connect: connection refused"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.024486    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb" podUID="17267346-b1b3-475b-9d89-1fda6e36aab5" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": dial tcp 10.132.0.42:8081: connect: connection refused"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.029221    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.029269    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.029310    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.029342    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.029383    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.029429    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.029473    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.029519    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-46l5d"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.206295    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.208145    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-qmpgn"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.208203    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.208292    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.208322    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.208416    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-46l5d"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.224991    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.280080    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.331302    5070 generic.go:356] "Generic (PLEG): container finished" podID="0d26093c-a60a-4cd8-9d43-8ecded62d895" containerID="bfd14b3b1a48099d8a2943d58cb72793f8e8fd272d5702cec8dc340d02fd1f29" exitCode=0
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.347181    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw" event={"ID":"0d26093c-a60a-4cd8-9d43-8ecded62d895","Type":"ContainerDied","Data":"bfd14b3b1a48099d8a2943d58cb72793f8e8fd272d5702cec8dc340d02fd1f29"}
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.368187    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb" podUID="17267346-b1b3-475b-9d89-1fda6e36aab5" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/readyz\": dial tcp 10.132.0.42:8081: connect: connection refused"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.370708    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-79698c9d55-m2lbh"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.404874    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p"]
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.423056    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-228695-2td9p"]
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.494374    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.495475    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212c33bc-85a6-4ff5-a96a-eda0064fab63-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.500692    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69lxp\" (UniqueName: \"kubernetes.io/projected/198146ed-665b-4801-8396-c980ee6ff4d4-kube-api-access-69lxp\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.633826    5070 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-n28db container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.633910    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db" podUID="5738916f-8202-40ed-823e-90f7c9cc4ec9" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.633977    5070 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db"
Nov 03 08:17:07.947035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.874084    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.964637    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-istio-envoy\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.964908    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-token\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:07.965005    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wgkx\" (UniqueName: \"kubernetes.io/projected/212c33bc-85a6-4ff5-a96a-eda0064fab63-kube-api-access-2wgkx\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.002850    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m"]
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.024634    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-credential-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.024782    5070 generic.go:356] "Generic (PLEG): container finished" podID="22591a72-cf89-4406-8e20-82fd6d0f6e6a" containerID="d9fea37b92890625680a72e06e7f19262c344e2ce9b5941ca192b5f45e22ca33" exitCode=0
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.024825    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-podinfo\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.026440    5070 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="csi-driver" containerStatusID={"Type":"cri-o","ID":"37ae4836852f00cdf0378a242817556fd99fede4156d44b96ef0a27d41757bb2"} pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db" containerMessage="Container csi-driver failed liveness probe, will be restarted"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.029246    5070 patch_prober.go:29] interesting pod/console-58b4bd6669-gzcxn container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.132.0.36:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.029292    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-58b4bd6669-gzcxn" podUID="873e56e5-1f64-4eec-a578-bf4ddd072869" containerName="console" probeResult="failure" output="Get \"https://10.132.0.36:8443/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.042440    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-workload-certs\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.042641    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.042663    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istiod-ca-cert\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.042797    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b95ll\" (UniqueName: \"kubernetes.io/projected/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-kube-api-access-b95ll\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.043300    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="989ac148-0bab-4020-939d-e1d087b018a5" path="/var/lib/kubelet/pods/989ac148-0bab-4020-939d-e1d087b018a5/volumes"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.043790    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58b4bd6669-gzcxn"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.044036    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j" event={"ID":"22591a72-cf89-4406-8e20-82fd6d0f6e6a","Type":"ContainerDied","Data":"d9fea37b92890625680a72e06e7f19262c344e2ce9b5941ca192b5f45e22ca33"}
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.063889    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.063906    5070 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-n5bqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.063926    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.063948    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg" podUID="63d796bd-8897-4067-b1b4-e39aecd06b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.119382    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/198146ed-665b-4801-8396-c980ee6ff4d4-istio-token\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.120787    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/198146ed-665b-4801-8396-c980ee6ff4d4-istio-podinfo\") pod \"e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") " pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.145214    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.146078    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.147051    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.184097    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.193977    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-data\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.196859    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-workload-socket\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.199252    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-envoy\") pod \"e2e-api-b-228695-openshift-default-547df686cf-62d7m\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") " pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.272431    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.273018    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db" podUID="5738916f-8202-40ed-823e-90f7c9cc4ec9" containerName="csi-driver" containerID="cri-o://37ae4836852f00cdf0378a242817556fd99fede4156d44b96ef0a27d41757bb2" gracePeriod=30
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.273271    5070 generic.go:356] "Generic (PLEG): container finished" podID="17267346-b1b3-475b-9d89-1fda6e36aab5" containerID="433fb34ce9e3ab80ce21604126a5f9e2b68325fb9f3a1f7aeb7694fbc95ec2db" exitCode=1
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.276438    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb" event={"ID":"17267346-b1b3-475b-9d89-1fda6e36aab5","Type":"ContainerDied","Data":"433fb34ce9e3ab80ce21604126a5f9e2b68325fb9f3a1f7aeb7694fbc95ec2db"}
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.276852    5070 scope.go:122] "RemoveContainer" containerID="433fb34ce9e3ab80ce21604126a5f9e2b68325fb9f3a1f7aeb7694fbc95ec2db"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.325212    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.369881    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:08.369928    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:09.105877    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:09.105939    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:09.105889    5070 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-n5bqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:09.106007    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg" podUID="63d796bd-8897-4067-b1b4-e39aecd06b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:09.383831    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-n28db_5738916f-8202-40ed-823e-90f7c9cc4ec9/csi-driver/0.log"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:09.383902    5070 generic.go:356] "Generic (PLEG): container finished" podID="5738916f-8202-40ed-823e-90f7c9cc4ec9" containerID="37ae4836852f00cdf0378a242817556fd99fede4156d44b96ef0a27d41757bb2" exitCode=2
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:09.383964    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db" event={"ID":"5738916f-8202-40ed-823e-90f7c9cc4ec9","Type":"ContainerDied","Data":"37ae4836852f00cdf0378a242817556fd99fede4156d44b96ef0a27d41757bb2"}
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:09.454980    5070 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-5qpnb container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.18:8443/readyz\": dial tcp 10.132.0.18:8443: connect: connection refused" start-of-body=
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:09.455021    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" podUID="d83648de-81dd-4160-8fa9-06d36b38d9de" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/readyz\": dial tcp 10.132.0.18:8443: connect: connection refused"
Nov 03 08:17:09.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:09.454981    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw" podUID="0d26093c-a60a-4cd8-9d43-8ecded62d895" containerName="server" probeResult="failure" output="Get \"https://10.132.0.47:8443/readyz\": dial tcp 10.132.0.47:8443: connect: connection refused"
Nov 03 08:17:10.733789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:10.515426    5070 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb"
Nov 03 08:17:10.967834 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:17:10.647242    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f04061_9793_4d5f_8189_141d7a74d515.slice/crio-83610e16a881285c4bce913bc1838b09c72d36f85e48037e4092f713b2c03620 WatchSource:0}: Error finding container 83610e16a881285c4bce913bc1838b09c72d36f85e48037e4092f713b2c03620: Status 404 returned error can't find the container with id 83610e16a881285c4bce913bc1838b09c72d36f85e48037e4092f713b2c03620
Nov 03 08:17:11.422871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:11.422835    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m" event={"ID":"30f04061-9793-4d5f-8189-141d7a74d515","Type":"ContainerStarted","Data":"83610e16a881285c4bce913bc1838b09c72d36f85e48037e4092f713b2c03620"}
Nov 03 08:17:12.228205 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:12.122025    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/3.log"
Nov 03 08:17:13.048059 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:13.047984    5070 trace.go:236] Trace[1855956282]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65" (03-Nov-2026 08:17:11.702) (total time: 1345ms):
Nov 03 08:17:13.048059 ip-10-0-140-123 kubenswrapper[5070]: Trace[1855956282]: [1.345452802s] [1.345452802s] END
Nov 03 08:17:13.376040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:13.296129    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/3.log"
Nov 03 08:17:13.376040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:13.296333    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" event={"ID":"d83648de-81dd-4160-8fa9-06d36b38d9de","Type":"ContainerStarted","Data":"be7bd858acdc53288040f61de70c6b6c42d2c72243e4c1fecb92dea8dc03b140"}
Nov 03 08:17:13.376040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:13.300805    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j" event={"ID":"22591a72-cf89-4406-8e20-82fd6d0f6e6a","Type":"ContainerStarted","Data":"182b3caa7de417605d5defd37ebf15caf24c5e3da0c66d78186d867909b5fdc3"}
Nov 03 08:17:13.376040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:13.301353    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j"
Nov 03 08:17:13.376040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:13.301451    5070 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-j5h2j container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": dial tcp 10.132.0.30:8443: connect: connection refused" start-of-body=
Nov 03 08:17:13.376040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:13.301499    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j" podUID="22591a72-cf89-4406-8e20-82fd6d0f6e6a" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": dial tcp 10.132.0.30:8443: connect: connection refused"
Nov 03 08:17:13.376040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:13.307207    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw" event={"ID":"0d26093c-a60a-4cd8-9d43-8ecded62d895","Type":"ContainerStarted","Data":"b5f838be0eac222bbd0902f601aaec6e39112af6c12e7711db6fb41cc233e0db"}
Nov 03 08:17:13.376040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:13.308096    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw"
Nov 03 08:17:14.345379 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:14.345339    5070 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-5qpnb container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.18:8443/readyz\": dial tcp 10.132.0.18:8443: connect: connection refused" start-of-body=
Nov 03 08:17:14.345379 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:14.345400    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" podUID="d83648de-81dd-4160-8fa9-06d36b38d9de" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/readyz\": dial tcp 10.132.0.18:8443: connect: connection refused"
Nov 03 08:17:14.394049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:14.345895    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb" event={"ID":"17267346-b1b3-475b-9d89-1fda6e36aab5","Type":"ContainerStarted","Data":"08458a97c8e7baf449a59347880c899c4303e49539deb8c2ce4f202f879b4f8f"}
Nov 03 08:17:14.394049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:14.345955    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 08:17:14.394049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:14.345975    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb"
Nov 03 08:17:14.394049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:14.346535    5070 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-j5h2j container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.132.0.30:8443/healthz\": dial tcp 10.132.0.30:8443: connect: connection refused" start-of-body=
Nov 03 08:17:14.394049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:14.346572    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j" podUID="22591a72-cf89-4406-8e20-82fd6d0f6e6a" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.30:8443/healthz\": dial tcp 10.132.0.30:8443: connect: connection refused"
Nov 03 08:17:15.323556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:15.323478    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"]
Nov 03 08:17:15.338635 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:15.338603    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"]
Nov 03 08:17:15.346729 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:15.346697    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"]
Nov 03 08:17:15.355976 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:15.355945    5070 generic.go:356] "Generic (PLEG): container finished" podID="9ecd3f55-84e0-4321-8d19-900388b6a91f" containerID="da923a33ad03f3e4cbb7235fcb0dbe82c426d767fa9a0045b6aacb3a97ac314a" exitCode=1
Nov 03 08:17:15.356113 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:15.356014    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65" event={"ID":"9ecd3f55-84e0-4321-8d19-900388b6a91f","Type":"ContainerDied","Data":"da923a33ad03f3e4cbb7235fcb0dbe82c426d767fa9a0045b6aacb3a97ac314a"}
Nov 03 08:17:15.357580 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:15.357560    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m" event={"ID":"30f04061-9793-4d5f-8189-141d7a74d515","Type":"ContainerStarted","Data":"aeab18f2416d23fc7f4c44c3bc5a7381d990c8c3f40204fbc8b7e38835f8a4dc"}
Nov 03 08:17:15.677511 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:15.677138    5070 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-5qpnb container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.18:8443/healthz\": dial tcp 10.132.0.18:8443: connect: connection refused" start-of-body=
Nov 03 08:17:15.677511 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:15.677195    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" podUID="d83648de-81dd-4160-8fa9-06d36b38d9de" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/healthz\": dial tcp 10.132.0.18:8443: connect: connection refused"
Nov 03 08:17:15.702828 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:15.697978    5070 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 08:17:15.760777 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:15.749036    5070 scope.go:122] "RemoveContainer" containerID="da923a33ad03f3e4cbb7235fcb0dbe82c426d767fa9a0045b6aacb3a97ac314a"
Nov 03 08:17:15.774783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:15.770556    5070 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-5qpnb container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.18:8443/readyz\": dial tcp 10.132.0.18:8443: connect: connection refused" start-of-body=
Nov 03 08:17:15.774783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:15.770609    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" podUID="d83648de-81dd-4160-8fa9-06d36b38d9de" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/readyz\": dial tcp 10.132.0.18:8443: connect: connection refused"
Nov 03 08:17:15.866964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:15.851254    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m" podStartSLOduration=57.851235673 podStartE2EDuration="57.851235673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:15.837528909 +0000 UTC m=+1433.951992840" watchObservedRunningTime="2026-11-03 08:17:15.851235673 +0000 UTC m=+1433.965699605"
Nov 03 08:17:15.973728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:15.973587    5070 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 08:17:16.064855 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:16.061945    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg" podUID="63d796bd-8897-4067-b1b4-e39aecd06b88" containerName="istio-proxy" containerID="cri-o://b11717b193b9d6894c25850a626a8a1063989c6c412a8f4f6606a2e54788de64" gracePeriod=30
Nov 03 08:17:16.343231 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:16.343132    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657" podUID="870ca6ae-460c-4c9e-9d51-5140a7025bc5" containerName="istio-proxy" containerID="cri-o://283a6618d7b74aa3c22c7e8daea17b88f1a06ebefb40f4b446c0be1e4beeabc2" gracePeriod=30
Nov 03 08:17:16.562018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:16.561980    5070 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-5qpnb container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.18:8443/readyz\": dial tcp 10.132.0.18:8443: connect: connection refused" start-of-body=
Nov 03 08:17:16.672457 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:16.562057    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb" podUID="d83648de-81dd-4160-8fa9-06d36b38d9de" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.18:8443/readyz\": dial tcp 10.132.0.18:8443: connect: connection refused"
Nov 03 08:17:16.938129 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:16.820549    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"]
Nov 03 08:17:16.938129 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:16.820895    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r" podUID="181a2d26-3317-450e-97be-c0082d47a27a" containerName="istio-proxy" containerID="cri-o://c96a23ab691a87d470784c9aa049512543d087810e8fc7695666836b49509f61" gracePeriod=30
Nov 03 08:17:16.938129 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:16.832935    5070 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-n5bqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:17:16.938129 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:16.832983    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg" podUID="63d796bd-8897-4067-b1b4-e39aecd06b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:17.053689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:16.997851    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"]
Nov 03 08:17:17.053689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:16.998422    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q" podUID="b1cb7af3-82c2-482c-aa2d-510c50735024" containerName="istio-proxy" containerID="cri-o://2e46fc0ae212af5ae9cfd6003a9203d3f9bc74178aeb56ad9dbc7926011bbb5c" gracePeriod=30
Nov 03 08:17:17.761661 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:17.178001    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"]
Nov 03 08:17:17.761661 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:17.178201    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs" podUID="2abf1628-8c68-450c-a458-eefa78b239b3" containerName="istio-proxy" containerID="cri-o://d93bb40e960d77605b81abd3b9d587c894c99bcfd66e65d1133c662cfa9e242b" gracePeriod=30
Nov 03 08:17:17.761661 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:17.223612    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"]
Nov 03 08:17:17.761661 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:17.223875    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy" containerID="cri-o://1fbbaf459fb90ee32385f9047f532e8ad7c2e1957187ff3c870517e885d4090a" gracePeriod=30
Nov 03 08:17:17.761661 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:17.252752    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"]
Nov 03 08:17:17.761661 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:17.252990    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6" podUID="7f0c52e6-b384-411f-9c3a-c2d76c149238" containerName="istio-proxy" containerID="cri-o://e719b3e626dc6538cd9eca46d8bad4fa011b80318345a2b254b98549ea65baf1" gracePeriod=30
Nov 03 08:17:17.761661 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:17.632164    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j" podUID="67eb2350-8895-4e7d-b1f2-e78e1ff5cf23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:19.926663    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:19.926830    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm" podUID="5e519992-15af-4d72-ba82-190c90ab0aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.621816    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"]
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.626501    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"]
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.631503    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"]
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:17:20.802997    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198146ed_665b_4801_8396_c980ee6ff4d4.slice/crio-261a8a38390427188cd605257dde1373746ce43954264f998c8fbf17b87043c5 WatchSource:0}: Error finding container 261a8a38390427188cd605257dde1373746ce43954264f998c8fbf17b87043c5: Status 404 returned error can't find the container with id 261a8a38390427188cd605257dde1373746ce43954264f998c8fbf17b87043c5
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.868978    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.868978    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.868978    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.883621    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m" event={"ID":"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede","Type":"ContainerStarted","Data":"aee61dbba7ee551a74c72f15af2acd7f09833ad87b1e2ae3ffb6e39daa165c91"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.885177    5070 generic.go:356] "Generic (PLEG): container finished" podID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerID="5c89f506e79ac82b260040c7148b1e731c64138a5485a246e28a4ad6a3fb01d1" exitCode=0
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.885237    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" event={"ID":"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6","Type":"ContainerDied","Data":"5c89f506e79ac82b260040c7148b1e731c64138a5485a246e28a4ad6a3fb01d1"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.885697    5070 scope.go:122] "RemoveContainer" containerID="5c89f506e79ac82b260040c7148b1e731c64138a5485a246e28a4ad6a3fb01d1"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.886533    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk" event={"ID":"212c33bc-85a6-4ff5-a96a-eda0064fab63","Type":"ContainerStarted","Data":"30ee49bd3e0fd4d1b007c0abb82587e4ef0ca53e9f8cbbc69d603aa08e0d4c71"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.887599    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.888005    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv" event={"ID":"198146ed-665b-4801-8396-c980ee6ff4d4","Type":"ContainerStarted","Data":"261a8a38390427188cd605257dde1373746ce43954264f998c8fbf17b87043c5"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.889609    5070 generic.go:356] "Generic (PLEG): container finished" podID="30f04061-9793-4d5f-8189-141d7a74d515" containerID="aeab18f2416d23fc7f4c44c3bc5a7381d990c8c3f40204fbc8b7e38835f8a4dc" exitCode=0
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.889645    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m" event={"ID":"30f04061-9793-4d5f-8189-141d7a74d515","Type":"ContainerDied","Data":"aeab18f2416d23fc7f4c44c3bc5a7381d990c8c3f40204fbc8b7e38835f8a4dc"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.907120    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.907124    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.907150    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.907174    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.907202    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.907125    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.907290    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:20.911824    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:21.319173    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:21.319241    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:21.643936    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:17:22.746810    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181a2d26_3317_450e_97be_c0082d47a27a.slice/crio-conmon-c96a23ab691a87d470784c9aa049512543d087810e8fc7695666836b49509f61.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abf1628_8c68_450c_a458_eefa78b239b3.slice/crio-d93bb40e960d77605b81abd3b9d587c894c99bcfd66e65d1133c662cfa9e242b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cb7af3_82c2_482c_aa2d_510c50735024.slice/crio-2e46fc0ae212af5ae9cfd6003a9203d3f9bc74178aeb56ad9dbc7926011bbb5c.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:22.920605    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m" event={"ID":"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede","Type":"ContainerStarted","Data":"3667ff4c14a711e5b3816efb81e3455342d1d124d7093301d42af5b12378791e"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:22.920578    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m" podUID="3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede" containerName="istio-proxy" containerID="cri-o://3667ff4c14a711e5b3816efb81e3455342d1d124d7093301d42af5b12378791e" gracePeriod=30
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:22.999573    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" event={"ID":"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6","Type":"ContainerStarted","Data":"af8c47943e052cd3b42beaabdc4afa3e932d82b9358e85fdb77720ea38f9921c"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.003152    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk" event={"ID":"212c33bc-85a6-4ff5-a96a-eda0064fab63","Type":"ContainerStarted","Data":"906ca3f54a6077a62a9475c973ab73f82b8058b800fa080208b163d7c16361e5"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.005481    5070 generic.go:356] "Generic (PLEG): container finished" podID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerID="1fbbaf459fb90ee32385f9047f532e8ad7c2e1957187ff3c870517e885d4090a" exitCode=0
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.005579    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" event={"ID":"17e8f246-f7e4-4dfa-afef-0fb361672461","Type":"ContainerDied","Data":"1fbbaf459fb90ee32385f9047f532e8ad7c2e1957187ff3c870517e885d4090a"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.007750    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv" event={"ID":"198146ed-665b-4801-8396-c980ee6ff4d4","Type":"ContainerStarted","Data":"aaba34aa082b19fa1fe5fd4bf0982a15a6b4e7a9fd5623ca022458171a88a036"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.010401    5070 generic.go:356] "Generic (PLEG): container finished" podID="2abf1628-8c68-450c-a458-eefa78b239b3" containerID="d93bb40e960d77605b81abd3b9d587c894c99bcfd66e65d1133c662cfa9e242b" exitCode=0
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.010474    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs" event={"ID":"2abf1628-8c68-450c-a458-eefa78b239b3","Type":"ContainerDied","Data":"d93bb40e960d77605b81abd3b9d587c894c99bcfd66e65d1133c662cfa9e242b"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.026349    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.027193    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.027238    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.030532    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m" podStartSLOduration=53.030514186 podStartE2EDuration="53.030514186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:23.014942949 +0000 UTC m=+1441.129406880" watchObservedRunningTime="2026-11-03 08:17:23.030514186 +0000 UTC m=+1441.144978118"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.039036    5070 generic.go:356] "Generic (PLEG): container finished" podID="181a2d26-3317-450e-97be-c0082d47a27a" containerID="c96a23ab691a87d470784c9aa049512543d087810e8fc7695666836b49509f61" exitCode=0
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.039081    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r" event={"ID":"181a2d26-3317-450e-97be-c0082d47a27a","Type":"ContainerDied","Data":"c96a23ab691a87d470784c9aa049512543d087810e8fc7695666836b49509f61"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.050191    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk" podStartSLOduration=54.050175916 podStartE2EDuration="54.050175916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:23.045656633 +0000 UTC m=+1441.160120568" watchObservedRunningTime="2026-11-03 08:17:23.050175916 +0000 UTC m=+1441.164639848"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.110781    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv" podStartSLOduration=58.110734414 podStartE2EDuration="58.110734414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:23.082030453 +0000 UTC m=+1441.196494386" watchObservedRunningTime="2026-11-03 08:17:23.110734414 +0000 UTC m=+1441.225198347"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.195572    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.196272    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.196326    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv" podUID="198146ed-665b-4801-8396-c980ee6ff4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.201986    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:17:23.216386    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.272665    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.328325    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.328401    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.333130    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.333195    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk" podUID="212c33bc-85a6-4ff5-a96a-eda0064fab63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.346059    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.550845    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30f04061-9793-4d5f-8189-141d7a74d515-kube-api-access-x2z2v\" (UniqueName: \"kubernetes.io/projected/30f04061-9793-4d5f-8189-141d7a74d515-kube-api-access-x2z2v\") pod \"30f04061-9793-4d5f-8189-141d7a74d515\" (UID: \"30f04061-9793-4d5f-8189-141d7a74d515\") "
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.550959    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30f04061-9793-4d5f-8189-141d7a74d515-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f04061-9793-4d5f-8189-141d7a74d515-maas-api-openshift-ca\") pod \"30f04061-9793-4d5f-8189-141d7a74d515\" (UID: \"30f04061-9793-4d5f-8189-141d7a74d515\") "
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.551058    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30f04061-9793-4d5f-8189-141d7a74d515-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f04061-9793-4d5f-8189-141d7a74d515-maas-api-xks-ca\") pod \"30f04061-9793-4d5f-8189-141d7a74d515\" (UID: \"30f04061-9793-4d5f-8189-141d7a74d515\") "
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.571424    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f04061-9793-4d5f-8189-141d7a74d515-maas-api-openshift-ca" pod "30f04061-9793-4d5f-8189-141d7a74d515" (UID: "30f04061-9793-4d5f-8189-141d7a74d515"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.833751    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30f04061-9793-4d5f-8189-141d7a74d515-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.917001    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30f04061-9793-4d5f-8189-141d7a74d515-kube-api-access-x2z2v" pod "30f04061-9793-4d5f-8189-141d7a74d515" (UID: "30f04061-9793-4d5f-8189-141d7a74d515"). InnerVolumeSpecName "kube-api-access-x2z2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.917505    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30f04061-9793-4d5f-8189-141d7a74d515-maas-api-xks-ca" pod "30f04061-9793-4d5f-8189-141d7a74d515" (UID: "30f04061-9793-4d5f-8189-141d7a74d515"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.945967    5070 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-hq657 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.946016    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657" podUID="870ca6ae-460c-4c9e-9d51-5140a7025bc5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.964866    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2z2v\" (UniqueName: \"kubernetes.io/projected/30f04061-9793-4d5f-8189-141d7a74d515-kube-api-access-x2z2v\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:23.964900    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30f04061-9793-4d5f-8189-141d7a74d515-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.060073    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.060129    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.132222    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.141086    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.141147    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.154594    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.206509    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.206587    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv" podUID="198146ed-665b-4801-8396-c980ee6ff4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.210649    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-token\" (UniqueName: \"kubernetes.io/projected/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-token\") pod \"17e8f246-f7e4-4dfa-afef-0fb361672461\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") "
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.210726    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-data\") pod \"17e8f246-f7e4-4dfa-afef-0fb361672461\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") "
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.210784    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m" event={"ID":"30f04061-9793-4d5f-8189-141d7a74d515","Type":"ContainerDied","Data":"83610e16a881285c4bce913bc1838b09c72d36f85e48037e4092f713b2c03620"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.210818    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-workload-certs\") pod \"17e8f246-f7e4-4dfa-afef-0fb361672461\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") "
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.210847    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83610e16a881285c4bce913bc1838b09c72d36f85e48037e4092f713b2c03620"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.210860    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-workload-socket\") pod \"17e8f246-f7e4-4dfa-afef-0fb361672461\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") "
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.210921    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-podinfo\") pod \"17e8f246-f7e4-4dfa-afef-0fb361672461\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") "
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.211033    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e8f246-f7e4-4dfa-afef-0fb361672461-kube-api-access-wszhb\" (UniqueName: \"kubernetes.io/projected/17e8f246-f7e4-4dfa-afef-0fb361672461-kube-api-access-wszhb\") pod \"17e8f246-f7e4-4dfa-afef-0fb361672461\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") "
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.211103    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-credential-socket\") pod \"17e8f246-f7e4-4dfa-afef-0fb361672461\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") "
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.211195    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-envoy\") pod \"17e8f246-f7e4-4dfa-afef-0fb361672461\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") "
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.211491    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e8f246-f7e4-4dfa-afef-0fb361672461-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e8f246-f7e4-4dfa-afef-0fb361672461-istiod-ca-cert\") pod \"17e8f246-f7e4-4dfa-afef-0fb361672461\" (UID: \"17e8f246-f7e4-4dfa-afef-0fb361672461\") "
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.212730    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e8f246-f7e4-4dfa-afef-0fb361672461-istiod-ca-cert" pod "17e8f246-f7e4-4dfa-afef-0fb361672461" (UID: "17e8f246-f7e4-4dfa-afef-0fb361672461"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.237084    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-workload-certs" pod "17e8f246-f7e4-4dfa-afef-0fb361672461" (UID: "17e8f246-f7e4-4dfa-afef-0fb361672461"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.245060    5070 patch_prober.go:29] interesting pod/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.245112    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs" podUID="2abf1628-8c68-450c-a458-eefa78b239b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.249450    5070 generic.go:356] "Generic (PLEG): container finished" podID="b1cb7af3-82c2-482c-aa2d-510c50735024" containerID="2e46fc0ae212af5ae9cfd6003a9203d3f9bc74178aeb56ad9dbc7926011bbb5c" exitCode=0
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.249639    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q" event={"ID":"b1cb7af3-82c2-482c-aa2d-510c50735024","Type":"ContainerDied","Data":"2e46fc0ae212af5ae9cfd6003a9203d3f9bc74178aeb56ad9dbc7926011bbb5c"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.254590    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"]
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.255973    5070 cpu_manager.go:414] "containerMap: removing container" podUID="30f04061-9793-4d5f-8189-141d7a74d515" containerName="revoke-keys"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.284358    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30f04061-9793-4d5f-8189-141d7a74d515" containerName="revoke-keys"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.284534    5070 cpu_manager.go:414] "containerMap: removing container" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.284545    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.284891    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="30f04061-9793-4d5f-8189-141d7a74d515" containerName="revoke-keys"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.284918    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" containerName="istio-proxy"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.306267    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-credential-socket" pod "17e8f246-f7e4-4dfa-afef-0fb361672461" (UID: "17e8f246-f7e4-4dfa-afef-0fb361672461"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.342900    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.342959    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk" podUID="212c33bc-85a6-4ff5-a96a-eda0064fab63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.349348    5070 generic.go:356] "Generic (PLEG): container finished" podID="870ca6ae-460c-4c9e-9d51-5140a7025bc5" containerID="283a6618d7b74aa3c22c7e8daea17b88f1a06ebefb40f4b446c0be1e4beeabc2" exitCode=0
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.350188    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e8f246-f7e4-4dfa-afef-0fb361672461-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.350220    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.350236    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.384283    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.386422    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-workload-socket" pod "17e8f246-f7e4-4dfa-afef-0fb361672461" (UID: "17e8f246-f7e4-4dfa-afef-0fb361672461"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.386697    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-data" pod "17e8f246-f7e4-4dfa-afef-0fb361672461" (UID: "17e8f246-f7e4-4dfa-afef-0fb361672461"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.404655    5070 generic.go:356] "Generic (PLEG): container finished" podID="7f0c52e6-b384-411f-9c3a-c2d76c149238" containerID="e719b3e626dc6538cd9eca46d8bad4fa011b80318345a2b254b98549ea65baf1" exitCode=0
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.512525    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.512555    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.611134    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e8f246-f7e4-4dfa-afef-0fb361672461-kube-api-access-wszhb" pod "17e8f246-f7e4-4dfa-afef-0fb361672461" (UID: "17e8f246-f7e4-4dfa-afef-0fb361672461"). InnerVolumeSpecName "kube-api-access-wszhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.611220    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657" event={"ID":"870ca6ae-460c-4c9e-9d51-5140a7025bc5","Type":"ContainerDied","Data":"283a6618d7b74aa3c22c7e8daea17b88f1a06ebefb40f4b446c0be1e4beeabc2"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.611281    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-token" pod "17e8f246-f7e4-4dfa-afef-0fb361672461" (UID: "17e8f246-f7e4-4dfa-afef-0fb361672461"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.611313    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-j5h2j"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.611347    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7" event={"ID":"17e8f246-f7e4-4dfa-afef-0fb361672461","Type":"ContainerDied","Data":"a0377f95201a4ae5a147443f9c4adb873197980fa47a2855efcb47ceba51b406"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.611375    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6" event={"ID":"7f0c52e6-b384-411f-9c3a-c2d76c149238","Type":"ContainerDied","Data":"e719b3e626dc6538cd9eca46d8bad4fa011b80318345a2b254b98549ea65baf1"}
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.611387    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.611408    5070 scope.go:122] "RemoveContainer" containerID="1fbbaf459fb90ee32385f9047f532e8ad7c2e1957187ff3c870517e885d4090a"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.615104    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-workload-socket\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.615150    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-data\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.615240    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-workload-certs\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.615308    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-podinfo\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.615358    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/312a9c42-79df-4b2d-b430-0c02d7c37200-istiod-ca-cert\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.615443    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-credential-socket\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.615542    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-token\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.615574    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-envoy\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.615608    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmkng\" (UniqueName: \"kubernetes.io/projected/312a9c42-79df-4b2d-b430-0c02d7c37200-kube-api-access-nmkng\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.615749    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.615783    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wszhb\" (UniqueName: \"kubernetes.io/projected/17e8f246-f7e4-4dfa-afef-0fb361672461-kube-api-access-wszhb\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.617471    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-envoy" pod "17e8f246-f7e4-4dfa-afef-0fb361672461" (UID: "17e8f246-f7e4-4dfa-afef-0fb361672461"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.620866    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0c8a52-openshift-default-dockercfg-l65b4\""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.808469    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-workload-socket\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.808556    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-data\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.808624    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-workload-certs\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.808714    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-podinfo\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.808775    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/312a9c42-79df-4b2d-b430-0c02d7c37200-istiod-ca-cert\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.808890    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-credential-socket\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.809010    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-token\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.809040    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-envoy\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.809075    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmkng\" (UniqueName: \"kubernetes.io/projected/312a9c42-79df-4b2d-b430-0c02d7c37200-kube-api-access-nmkng\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.809203    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.846448    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-workload-socket\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.847630    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/312a9c42-79df-4b2d-b430-0c02d7c37200-istiod-ca-cert\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.847876    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-credential-socket\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.848600    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-workload-certs\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.852072    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"]
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.873058    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-podinfo" pod "17e8f246-f7e4-4dfa-afef-0fb361672461" (UID: "17e8f246-f7e4-4dfa-afef-0fb361672461"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:17:25.057547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:24.914819    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e8f246-f7e4-4dfa-afef-0fb361672461-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:25.105617    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:25.105675    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:25.269585    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:25.269642    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv" podUID="198146ed-665b-4801-8396-c980ee6ff4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:25.332476    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:25.335063    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:25.335107    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk" podUID="212c33bc-85a6-4ff5-a96a-eda0064fab63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:25.475388    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:25.488180    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:25.643879    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:25.644272    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:25.744850    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:25.769887    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:25.787753    5070 patch_prober.go:29] interesting pod/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:25.787827    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6" podUID="7f0c52e6-b384-411f-9c3a-c2d76c149238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:25.789196    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-data\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:25.805684    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-6kbrb"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:25.899298    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-mdwfw"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.014674    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"]
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.171714    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.171794    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.202825    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.235227    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.235283    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv" podUID="198146ed-665b-4801-8396-c980ee6ff4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.327299    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.327363    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk" podUID="212c33bc-85a6-4ff5-a96a-eda0064fab63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.338116    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-4nrz7"]
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.446398    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-podinfo\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.486103    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.508831    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.509777    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.556622    5070 patch_prober.go:29] interesting pod/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.556681    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r" podUID="181a2d26-3317-450e-97be-c0082d47a27a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.556983    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": context deadline exceeded"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.653634    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.661597    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.662626    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-token\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.670133    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-5qpnb"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.778573    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmkng\" (UniqueName: \"kubernetes.io/projected/312a9c42-79df-4b2d-b430-0c02d7c37200-kube-api-access-nmkng\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.792010    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.809969    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:26.810444    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.085084    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-envoy\") pod \"e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") " pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.085598    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.139638    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.139697    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.241706    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.243452    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.266806    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.266867    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv" podUID="198146ed-665b-4801-8396-c980ee6ff4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.267217    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.552030    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.552396    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:17:27.605288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.552608    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:17:28.053653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.709704    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:17:28.053653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.735618    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 08:17:28.053653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.735686    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk" podUID="212c33bc-85a6-4ff5-a96a-eda0064fab63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 08:17:28.053653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.820436    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:17:28.053653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.865642    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:28.053653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.871899    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:17:28.053653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.871945    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6" event={"ID":"7f0c52e6-b384-411f-9c3a-c2d76c149238","Type":"ContainerDied","Data":"f03129ecc759485af14ce97800b02658cab12775680741e7d372ff3eda0e31dd"}
Nov 03 08:17:28.053653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.871997    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f03129ecc759485af14ce97800b02658cab12775680741e7d372ff3eda0e31dd"
Nov 03 08:17:28.053653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.872362    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:17:28.053653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.916951    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs" event={"ID":"2abf1628-8c68-450c-a458-eefa78b239b3","Type":"ContainerDied","Data":"6cdce6e38e494579f85911a169a6b02f68662d54ad1c4c78be24bb2794097def"}
Nov 03 08:17:28.053653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.917010    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cdce6e38e494579f85911a169a6b02f68662d54ad1c4c78be24bb2794097def"
Nov 03 08:17:28.053653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.919195    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r" event={"ID":"181a2d26-3317-450e-97be-c0082d47a27a","Type":"ContainerDied","Data":"f77b03bc766f4214fe8c69fc350806398936e1d1b53358fc8863bca4d8709370"}
Nov 03 08:17:28.053653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.919247    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f77b03bc766f4214fe8c69fc350806398936e1d1b53358fc8863bca4d8709370"
Nov 03 08:17:28.053653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.921273    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q" event={"ID":"b1cb7af3-82c2-482c-aa2d-510c50735024","Type":"ContainerDied","Data":"5b51fda80ac1b6d422a0fdea95cba9a3493c298ee51393a15e091b7979b0f06a"}
Nov 03 08:17:28.053653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:27.921326    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b51fda80ac1b6d422a0fdea95cba9a3493c298ee51393a15e091b7979b0f06a"
Nov 03 08:17:28.428323 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:28.066051    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 08:17:28.428323 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:28.066124    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 08:17:28.685407 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:28.570924    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 03 08:17:28.685407 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:28.571013    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv" podUID="198146ed-665b-4801-8396-c980ee6ff4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 03 08:17:28.685407 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:28.622415    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 08:17:28.685407 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:28.622473    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk" podUID="212c33bc-85a6-4ff5-a96a-eda0064fab63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 08:17:28.714390 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:28.708729    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:28.749180 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:28.736358    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:28.749180 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:28.740860    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:28.848460 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:28.815361    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn" podUID="6960b432-0c5c-4c03-bcc5-dec9ff9466c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:28.909507 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:28.854842    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:28.909507 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:28.855572    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:17:28.968506 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:28.946070    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:17:28.968506 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:28.946479    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:17:29.022077 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.014527    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:17:29.118161 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.110846    5070 patch_prober.go:29] interesting pod/console-58b4bd6669-gzcxn container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.132.0.36:8443/health\": context deadline exceeded" start-of-body=
Nov 03 08:17:29.118161 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.110905    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/console-58b4bd6669-gzcxn" podUID="873e56e5-1f64-4eec-a578-bf4ddd072869" containerName="console" probeResult="failure" output="Get \"https://10.132.0.36:8443/health\": context deadline exceeded"
Nov 03 08:17:29.118161 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.114172    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e8f246-f7e4-4dfa-afef-0fb361672461" path="/var/lib/kubelet/pods/17e8f246-f7e4-4dfa-afef-0fb361672461/volumes"
Nov 03 08:17:29.118161 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:17:29.114719    5070 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.028s"
Nov 03 08:17:29.240390 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.226970    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:17:29.327897 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.327851    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j" podUID="67eb2350-8895-4e7d-b1f2-e78e1ff5cf23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded"
Nov 03 08:17:29.345207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.345166    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:17:29.497564 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.497451    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-workload-certs\") pod \"b1cb7af3-82c2-482c-aa2d-510c50735024\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") "
Nov 03 08:17:29.610931 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.610897    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1cb7af3-82c2-482c-aa2d-510c50735024-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1cb7af3-82c2-482c-aa2d-510c50735024-istiod-ca-cert\") pod \"b1cb7af3-82c2-482c-aa2d-510c50735024\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") "
Nov 03 08:17:30.034160 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.034126    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-workload-socket\") pod \"b1cb7af3-82c2-482c-aa2d-510c50735024\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") "
Nov 03 08:17:30.050944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.050896    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-credential-socket\") pod \"b1cb7af3-82c2-482c-aa2d-510c50735024\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") "
Nov 03 08:17:30.051285 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.051262    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-token\" (UniqueName: \"kubernetes.io/projected/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-token\") pod \"b1cb7af3-82c2-482c-aa2d-510c50735024\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") "
Nov 03 08:17:30.051455 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.051437    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1cb7af3-82c2-482c-aa2d-510c50735024-kube-api-access-4nf9m\" (UniqueName: \"kubernetes.io/projected/b1cb7af3-82c2-482c-aa2d-510c50735024-kube-api-access-4nf9m\") pod \"b1cb7af3-82c2-482c-aa2d-510c50735024\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") "
Nov 03 08:17:30.051636 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.051619    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-podinfo\") pod \"b1cb7af3-82c2-482c-aa2d-510c50735024\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") "
Nov 03 08:17:30.051863 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.051846    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-envoy\") pod \"b1cb7af3-82c2-482c-aa2d-510c50735024\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") "
Nov 03 08:17:30.052073 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.052060    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-data\") pod \"b1cb7af3-82c2-482c-aa2d-510c50735024\" (UID: \"b1cb7af3-82c2-482c-aa2d-510c50735024\") "
Nov 03 08:17:30.063677 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.756254    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:30.073825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.738884    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:30.112115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.940863    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:17:30.112467 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.112436    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:17:30.124921 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.563852    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 08:17:30.125201 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.125155    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk" podUID="212c33bc-85a6-4ff5-a96a-eda0064fab63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 08:17:30.128787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.603950    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-workload-certs" pod "b1cb7af3-82c2-482c-aa2d-510c50735024" (UID: "b1cb7af3-82c2-482c-aa2d-510c50735024"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:30.129000 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.739146    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1cb7af3-82c2-482c-aa2d-510c50735024-istiod-ca-cert" pod "b1cb7af3-82c2-482c-aa2d-510c50735024" (UID: "b1cb7af3-82c2-482c-aa2d-510c50735024"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:17:30.129104 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.848203    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 03 08:17:30.129261 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.129225    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv" podUID="198146ed-665b-4801-8396-c980ee6ff4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 03 08:17:30.129384 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.003180    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 08:17:30.129511 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.129493    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 08:17:30.155235 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.805150    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:30.182521 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.805089    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": context deadline exceeded"
Nov 03 08:17:30.190169 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.871984    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:17:30.232058 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.953217    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:30.236119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.236073    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:30.247648 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.503986    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:17:30.265797 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.965899    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:30.281821 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:29.904890    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:30.283088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.283058    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-data" pod "b1cb7af3-82c2-482c-aa2d-510c50735024" (UID: "b1cb7af3-82c2-482c-aa2d-510c50735024"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:30.286298 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.286228    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-credential-socket" pod "b1cb7af3-82c2-482c-aa2d-510c50735024" (UID: "b1cb7af3-82c2-482c-aa2d-510c50735024"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:30.317225 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.317168    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-workload-socket" pod "b1cb7af3-82c2-482c-aa2d-510c50735024" (UID: "b1cb7af3-82c2-482c-aa2d-510c50735024"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:30.354705 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.354669    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 08:17:30.354943 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.354915    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 08:17:30.357618 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.357595    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 03 08:17:30.357800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.357777    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv" podUID="198146ed-665b-4801-8396-c980ee6ff4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 03 08:17:30.372804 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.358386    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-podinfo" pod "b1cb7af3-82c2-482c-aa2d-510c50735024" (UID: "b1cb7af3-82c2-482c-aa2d-510c50735024"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:17:30.799878 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.540571    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 08:17:30.799878 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.699537    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk" podUID="212c33bc-85a6-4ff5-a96a-eda0064fab63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 08:17:30.799878 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.435607    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:30.799878 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.699609    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b1cb7af3-82c2-482c-aa2d-510c50735024-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:30.799878 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.629165    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:17:30.849952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.804632    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:17:30.849952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.808915    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-workload-socket\") pod \"181a2d26-3317-450e-97be-c0082d47a27a\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") "
Nov 03 08:17:30.849952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.809128    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-token\" (UniqueName: \"kubernetes.io/projected/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-token\") pod \"7f0c52e6-b384-411f-9c3a-c2d76c149238\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") "
Nov 03 08:17:30.849952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.809201    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-workload-certs\") pod \"7f0c52e6-b384-411f-9c3a-c2d76c149238\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") "
Nov 03 08:17:30.849952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.809275    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-istio-data\") pod \"181a2d26-3317-450e-97be-c0082d47a27a\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") "
Nov 03 08:17:30.849952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.809301    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-istio-envoy\") pod \"181a2d26-3317-450e-97be-c0082d47a27a\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") "
Nov 03 08:17:30.849952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.809374    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f0c52e6-b384-411f-9c3a-c2d76c149238-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0c52e6-b384-411f-9c3a-c2d76c149238-istiod-ca-cert\") pod \"7f0c52e6-b384-411f-9c3a-c2d76c149238\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") "
Nov 03 08:17:30.849952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.809425    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181a2d26-3317-450e-97be-c0082d47a27a-kube-api-access-cjlps\" (UniqueName: \"kubernetes.io/projected/181a2d26-3317-450e-97be-c0082d47a27a-kube-api-access-cjlps\") pod \"181a2d26-3317-450e-97be-c0082d47a27a\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") "
Nov 03 08:17:30.849952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.809458    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-credential-socket\") pod \"181a2d26-3317-450e-97be-c0082d47a27a\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") "
Nov 03 08:17:30.849952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.809498    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-envoy\") pod \"7f0c52e6-b384-411f-9c3a-c2d76c149238\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") "
Nov 03 08:17:30.849952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.809550    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-workload-socket\") pod \"7f0c52e6-b384-411f-9c3a-c2d76c149238\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") "
Nov 03 08:17:30.849952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.809579    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/181a2d26-3317-450e-97be-c0082d47a27a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181a2d26-3317-450e-97be-c0082d47a27a-istiod-ca-cert\") pod \"181a2d26-3317-450e-97be-c0082d47a27a\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") "
Nov 03 08:17:30.849952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.809636    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-podinfo\") pod \"7f0c52e6-b384-411f-9c3a-c2d76c149238\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") "
Nov 03 08:17:30.849952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.809664    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/181a2d26-3317-450e-97be-c0082d47a27a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181a2d26-3317-450e-97be-c0082d47a27a-istio-podinfo\") pod \"181a2d26-3317-450e-97be-c0082d47a27a\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") "
Nov 03 08:17:30.849952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.809774    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-workload-certs\") pod \"181a2d26-3317-450e-97be-c0082d47a27a\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") "
Nov 03 08:17:30.849952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.809867    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-data\") pod \"7f0c52e6-b384-411f-9c3a-c2d76c149238\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") "
Nov 03 08:17:30.849952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.809919    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/181a2d26-3317-450e-97be-c0082d47a27a-istio-token\" (UniqueName: \"kubernetes.io/projected/181a2d26-3317-450e-97be-c0082d47a27a-istio-token\") pod \"181a2d26-3317-450e-97be-c0082d47a27a\" (UID: \"181a2d26-3317-450e-97be-c0082d47a27a\") "
Nov 03 08:17:30.849952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.811039    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:30.849952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.811066    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:30.849952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.811081    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:30.849952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.811096    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:30.948720 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.923926    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-credential-socket\") pod \"7f0c52e6-b384-411f-9c3a-c2d76c149238\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") "
Nov 03 08:17:30.949102 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.949075    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0c52e6-b384-411f-9c3a-c2d76c149238-kube-api-access-jz2vf\" (UniqueName: \"kubernetes.io/projected/7f0c52e6-b384-411f-9c3a-c2d76c149238-kube-api-access-jz2vf\") pod \"7f0c52e6-b384-411f-9c3a-c2d76c149238\" (UID: \"7f0c52e6-b384-411f-9c3a-c2d76c149238\") "
Nov 03 08:17:30.950875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.933953    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:31.180239 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.075128    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:17:31.180239 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.075513    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:17:31.206982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.206936    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:31.300987 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.300954    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:17:31.345264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.328663    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:17:31.345264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.328907    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:17:31.345264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.329074    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:17:31.373235 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:17:31.364479    5070 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.252s"
Nov 03 08:17:31.373235 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.364567    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:17:31.373235 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.364597    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s"]
Nov 03 08:17:31.388074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.378662    5070 cpu_manager.go:414] "containerMap: removing container" podUID="181a2d26-3317-450e-97be-c0082d47a27a" containerName="istio-proxy"
Nov 03 08:17:31.388074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.378694    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="181a2d26-3317-450e-97be-c0082d47a27a" containerName="istio-proxy"
Nov 03 08:17:31.388074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.378722    5070 cpu_manager.go:414] "containerMap: removing container" podUID="7f0c52e6-b384-411f-9c3a-c2d76c149238" containerName="istio-proxy"
Nov 03 08:17:31.388074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.378730    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0c52e6-b384-411f-9c3a-c2d76c149238" containerName="istio-proxy"
Nov 03 08:17:31.388074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.378778    5070 cpu_manager.go:414] "containerMap: removing container" podUID="b1cb7af3-82c2-482c-aa2d-510c50735024" containerName="istio-proxy"
Nov 03 08:17:31.388074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.378786    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1cb7af3-82c2-482c-aa2d-510c50735024" containerName="istio-proxy"
Nov 03 08:17:31.388074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.378858    5070 cpu_manager.go:414] "containerMap: removing container" podUID="2abf1628-8c68-450c-a458-eefa78b239b3" containerName="istio-proxy"
Nov 03 08:17:31.388074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.378866    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2abf1628-8c68-450c-a458-eefa78b239b3" containerName="istio-proxy"
Nov 03 08:17:31.388074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.379041    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="2abf1628-8c68-450c-a458-eefa78b239b3" containerName="istio-proxy"
Nov 03 08:17:31.388074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.379056    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="181a2d26-3317-450e-97be-c0082d47a27a" containerName="istio-proxy"
Nov 03 08:17:31.388074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.379071    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0c52e6-b384-411f-9c3a-c2d76c149238" containerName="istio-proxy"
Nov 03 08:17:31.388074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.379086    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1cb7af3-82c2-482c-aa2d-510c50735024" containerName="istio-proxy"
Nov 03 08:17:31.433169 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.415432    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7d9549b467-hqhvz" podUID="5211e023-fa82-4010-939e-89afebb9e6b8" containerName="registry" containerID="cri-o://a0f5d44418a8ed4fdedf0b5f3452626063bda534bdd4c631195fca4a944673aa" gracePeriod=30
Nov 03 08:17:31.482024 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.473900    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:17:31.482024 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.477836    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s"]
Nov 03 08:17:31.482024 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.478030    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s"
Nov 03 08:17:31.623299 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.933609    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/181a2d26-3317-450e-97be-c0082d47a27a-istiod-ca-cert" pod "181a2d26-3317-450e-97be-c0082d47a27a" (UID: "181a2d26-3317-450e-97be-c0082d47a27a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:17:31.623549 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.958689    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-workload-socket" pod "181a2d26-3317-450e-97be-c0082d47a27a" (UID: "181a2d26-3317-450e-97be-c0082d47a27a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:31.623680 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:30.988467    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-istio-data" pod "181a2d26-3317-450e-97be-c0082d47a27a" (UID: "181a2d26-3317-450e-97be-c0082d47a27a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:31.625010 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.032928    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-workload-certs" pod "7f0c52e6-b384-411f-9c3a-c2d76c149238" (UID: "7f0c52e6-b384-411f-9c3a-c2d76c149238"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:31.626614 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.064933    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-workload-certs" pod "181a2d26-3317-450e-97be-c0082d47a27a" (UID: "181a2d26-3317-450e-97be-c0082d47a27a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:31.629874 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.079554    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-credential-socket" pod "7f0c52e6-b384-411f-9c3a-c2d76c149238" (UID: "7f0c52e6-b384-411f-9c3a-c2d76c149238"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:31.632325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.087077    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-data" pod "7f0c52e6-b384-411f-9c3a-c2d76c149238" (UID: "7f0c52e6-b384-411f-9c3a-c2d76c149238"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:31.634091 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.415631    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 08:17:31.745284 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.715508    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 08:17:31.745284 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.673406    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:17:31.795125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.627406    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a66d5c-dockercfg-glfnx\""
Nov 03 08:17:31.798317 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.798275    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 08:17:31.804062 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.803994    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk" podUID="212c33bc-85a6-4ff5-a96a-eda0064fab63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 08:17:31.811566 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.810241    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a66d5c\""
Nov 03 08:17:31.814599 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.814562    5070 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-n5bqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:17:31.956728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.956691    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg" podUID="63d796bd-8897-4067-b1b4-e39aecd06b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:31.956949 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.950386    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:17:31.962264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.801028    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:31.962264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.957740    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:31.962264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.957785    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:31.962264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.957800    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/181a2d26-3317-450e-97be-c0082d47a27a-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:31.962264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.957815    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:31.962264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.957832    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:31.962264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.957846    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:31.963519 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.963172    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0c52e6-b384-411f-9c3a-c2d76c149238-istiod-ca-cert" pod "7f0c52e6-b384-411f-9c3a-c2d76c149238" (UID: "7f0c52e6-b384-411f-9c3a-c2d76c149238"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:17:32.182367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.977003    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4" podUID="d50d86f3-08a1-419c-a77b-d7223e0567c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Nov 03 08:17:32.182367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:31.977791    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:17:32.182367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.004458    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4" podUID="d50d86f3-08a1-419c-a77b-d7223e0567c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:32.182367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.026898    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:17:32.182367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.063101    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-credential-socket" pod "181a2d26-3317-450e-97be-c0082d47a27a" (UID: "181a2d26-3317-450e-97be-c0082d47a27a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:32.182367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.090144    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:32.182367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.090567    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:17:32.182367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.113964    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-workload-socket" pod "7f0c52e6-b384-411f-9c3a-c2d76c149238" (UID: "7f0c52e6-b384-411f-9c3a-c2d76c149238"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.214464    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.214523    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv" podUID="198146ed-665b-4801-8396-c980ee6ff4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.215978    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused" start-of-body=
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.216019    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv" podUID="198146ed-665b-4801-8396-c980ee6ff4d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.201:15021/healthz/ready\": dial tcp 10.132.0.201:15021: connect: connection refused"
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.226259    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.226303    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.250636    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-istio-data\") pod \"2abf1628-8c68-450c-a458-eefa78b239b3\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") "
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.250745    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-workload-socket\") pod \"2abf1628-8c68-450c-a458-eefa78b239b3\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") "
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.250808    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-workload-socket\") pod \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") "
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.250872    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-data\") pod \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") "
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.250920    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istiod-ca-cert\") pod \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") "
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.251013    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-credential-socket\") pod \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") "
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.251064    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2abf1628-8c68-450c-a458-eefa78b239b3-istio-token\" (UniqueName: \"kubernetes.io/projected/2abf1628-8c68-450c-a458-eefa78b239b3-istio-token\") pod \"2abf1628-8c68-450c-a458-eefa78b239b3\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") "
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.251108    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-credential-socket\") pod \"2abf1628-8c68-450c-a458-eefa78b239b3\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") "
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.251181    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-istio-envoy\") pod \"2abf1628-8c68-450c-a458-eefa78b239b3\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") "
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.251212    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2abf1628-8c68-450c-a458-eefa78b239b3-kube-api-access-fhk94\" (UniqueName: \"kubernetes.io/projected/2abf1628-8c68-450c-a458-eefa78b239b3-kube-api-access-fhk94\") pod \"2abf1628-8c68-450c-a458-eefa78b239b3\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") "
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.251290    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-envoy\") pod \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") "
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.251379    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-workload-certs\") pod \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") "
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.251433    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2abf1628-8c68-450c-a458-eefa78b239b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abf1628-8c68-450c-a458-eefa78b239b3-istiod-ca-cert\") pod \"2abf1628-8c68-450c-a458-eefa78b239b3\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") "
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.251994    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84094812-6bd0-401a-a91c-d3d24e71501f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s\" (UID: \"84094812-6bd0-401a-a91c-d3d24e71501f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s"
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.252101    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpp2p\" (UniqueName: \"kubernetes.io/projected/84094812-6bd0-401a-a91c-d3d24e71501f-kube-api-access-kpp2p\") pod \"payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s\" (UID: \"84094812-6bd0-401a-a91c-d3d24e71501f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s"
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.252631    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0c52e6-b384-411f-9c3a-c2d76c149238-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.252654    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.252670    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.253560    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-workload-socket" pod "870ca6ae-460c-4c9e-9d51-5140a7025bc5" (UID: "870ca6ae-460c-4c9e-9d51-5140a7025bc5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.253816    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-data" pod "870ca6ae-460c-4c9e-9d51-5140a7025bc5" (UID: "870ca6ae-460c-4c9e-9d51-5140a7025bc5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.254180    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istiod-ca-cert" pod "870ca6ae-460c-4c9e-9d51-5140a7025bc5" (UID: "870ca6ae-460c-4c9e-9d51-5140a7025bc5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:17:32.288982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.254405    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-credential-socket" pod "870ca6ae-460c-4c9e-9d51-5140a7025bc5" (UID: "870ca6ae-460c-4c9e-9d51-5140a7025bc5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:32.362679 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.353493    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-workload-certs" pod "870ca6ae-460c-4c9e-9d51-5140a7025bc5" (UID: "870ca6ae-460c-4c9e-9d51-5140a7025bc5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:32.546219 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.534481    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1cb7af3-82c2-482c-aa2d-510c50735024-kube-api-access-4nf9m" pod "b1cb7af3-82c2-482c-aa2d-510c50735024" (UID: "b1cb7af3-82c2-482c-aa2d-510c50735024"). InnerVolumeSpecName "kube-api-access-4nf9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:32.546219 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.540895    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:17:32.555601 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.550276    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": context deadline exceeded"
Nov 03 08:17:32.555601 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.550629    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:32.555601 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.551012    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:17:32.627946 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.627907    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/181a2d26-3317-450e-97be-c0082d47a27a-istio-podinfo" pod "181a2d26-3317-450e-97be-c0082d47a27a" (UID: "181a2d26-3317-450e-97be-c0082d47a27a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:17:32.650751 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.650521    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2abf1628-8c68-450c-a458-eefa78b239b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abf1628-8c68-450c-a458-eefa78b239b3-istio-podinfo\") pod \"2abf1628-8c68-450c-a458-eefa78b239b3\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") "
Nov 03 08:17:32.650751 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.650573    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-token\" (UniqueName: \"kubernetes.io/projected/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-token\") pod \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") "
Nov 03 08:17:32.650751 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.650612    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-workload-certs\") pod \"2abf1628-8c68-450c-a458-eefa78b239b3\" (UID: \"2abf1628-8c68-450c-a458-eefa78b239b3\") "
Nov 03 08:17:32.650751 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.650660    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/870ca6ae-460c-4c9e-9d51-5140a7025bc5-kube-api-access-2rw5b\" (UniqueName: \"kubernetes.io/projected/870ca6ae-460c-4c9e-9d51-5140a7025bc5-kube-api-access-2rw5b\") pod \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") "
Nov 03 08:17:32.650751 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.650695    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-podinfo\") pod \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\" (UID: \"870ca6ae-460c-4c9e-9d51-5140a7025bc5\") "
Nov 03 08:17:32.673566 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.670936    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84094812-6bd0-401a-a91c-d3d24e71501f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s\" (UID: \"84094812-6bd0-401a-a91c-d3d24e71501f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s"
Nov 03 08:17:32.674120 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.674092    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpp2p\" (UniqueName: \"kubernetes.io/projected/84094812-6bd0-401a-a91c-d3d24e71501f-kube-api-access-kpp2p\") pod \"payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s\" (UID: \"84094812-6bd0-401a-a91c-d3d24e71501f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s"
Nov 03 08:17:32.680035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.677255    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:32.680376 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.680355    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:32.680511 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.680497    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/181a2d26-3317-450e-97be-c0082d47a27a-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:32.682904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.682886    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:32.683025 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.683012    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nf9m\" (UniqueName: \"kubernetes.io/projected/b1cb7af3-82c2-482c-aa2d-510c50735024-kube-api-access-4nf9m\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:32.683125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.683113    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:32.685941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.683207    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:32.970543 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:32.967245    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:33.040739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.040400    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:17:33.101274 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.096721    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:17:33.168012 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.101555    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:17:33.168012 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.130322    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-token" pod "b1cb7af3-82c2-482c-aa2d-510c50735024" (UID: "b1cb7af3-82c2-482c-aa2d-510c50735024"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:33.168012 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.130411    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-envoy" pod "b1cb7af3-82c2-482c-aa2d-510c50735024" (UID: "b1cb7af3-82c2-482c-aa2d-510c50735024"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:33.168012 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.130499    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-token" pod "7f0c52e6-b384-411f-9c3a-c2d76c149238" (UID: "7f0c52e6-b384-411f-9c3a-c2d76c149238"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:33.168012 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.130566    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-istio-envoy" pod "181a2d26-3317-450e-97be-c0082d47a27a" (UID: "181a2d26-3317-450e-97be-c0082d47a27a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:33.168012 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.130638    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181a2d26-3317-450e-97be-c0082d47a27a-kube-api-access-cjlps" pod "181a2d26-3317-450e-97be-c0082d47a27a" (UID: "181a2d26-3317-450e-97be-c0082d47a27a"). InnerVolumeSpecName "kube-api-access-cjlps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:33.168012 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.130700    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-podinfo" pod "7f0c52e6-b384-411f-9c3a-c2d76c149238" (UID: "7f0c52e6-b384-411f-9c3a-c2d76c149238"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:17:33.168012 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.153918    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0c52e6-b384-411f-9c3a-c2d76c149238-kube-api-access-jz2vf" pod "7f0c52e6-b384-411f-9c3a-c2d76c149238" (UID: "7f0c52e6-b384-411f-9c3a-c2d76c149238"). InnerVolumeSpecName "kube-api-access-jz2vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:33.168012 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.154076    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-envoy" pod "870ca6ae-460c-4c9e-9d51-5140a7025bc5" (UID: "870ca6ae-460c-4c9e-9d51-5140a7025bc5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:33.168012 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.154167    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 08:17:33.168012 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.154225    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk" podUID="212c33bc-85a6-4ff5-a96a-eda0064fab63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 08:17:33.168012 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.162608    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:33.168012 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.162848    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:33.168012 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.162961    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:33.273939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.207086    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.273939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.207123    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.273939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.207150    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/181a2d26-3317-450e-97be-c0082d47a27a-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.273939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.207166    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjlps\" (UniqueName: \"kubernetes.io/projected/181a2d26-3317-450e-97be-c0082d47a27a-kube-api-access-cjlps\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.273939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.207182    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.273939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.207197    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b1cb7af3-82c2-482c-aa2d-510c50735024-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.273939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.207216    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.273939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.207231    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz2vf\" (UniqueName: \"kubernetes.io/projected/7f0c52e6-b384-411f-9c3a-c2d76c149238-kube-api-access-jz2vf\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.273939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.229335    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body=
Nov 03 08:17:33.273939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.229400    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused"
Nov 03 08:17:33.273939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.247397    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84094812-6bd0-401a-a91c-d3d24e71501f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s\" (UID: \"84094812-6bd0-401a-a91c-d3d24e71501f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s"
Nov 03 08:17:33.273939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.252268    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2abf1628-8c68-450c-a458-eefa78b239b3-istiod-ca-cert" pod "2abf1628-8c68-450c-a458-eefa78b239b3" (UID: "2abf1628-8c68-450c-a458-eefa78b239b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:17:33.273939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.258221    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-workload-socket" pod "2abf1628-8c68-450c-a458-eefa78b239b3" (UID: "2abf1628-8c68-450c-a458-eefa78b239b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:33.273939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.262458    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/181a2d26-3317-450e-97be-c0082d47a27a-istio-token" pod "181a2d26-3317-450e-97be-c0082d47a27a" (UID: "181a2d26-3317-450e-97be-c0082d47a27a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:33.273939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.266821    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-envoy" pod "7f0c52e6-b384-411f-9c3a-c2d76c149238" (UID: "7f0c52e6-b384-411f-9c3a-c2d76c149238"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:33.273939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.266931    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-podinfo" pod "870ca6ae-460c-4c9e-9d51-5140a7025bc5" (UID: "870ca6ae-460c-4c9e-9d51-5140a7025bc5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:17:33.273939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.267045    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-token" pod "870ca6ae-460c-4c9e-9d51-5140a7025bc5" (UID: "870ca6ae-460c-4c9e-9d51-5140a7025bc5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:33.273939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.273582    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-workload-certs" pod "2abf1628-8c68-450c-a458-eefa78b239b3" (UID: "2abf1628-8c68-450c-a458-eefa78b239b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:33.287298 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.277898    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-credential-socket" pod "2abf1628-8c68-450c-a458-eefa78b239b3" (UID: "2abf1628-8c68-450c-a458-eefa78b239b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:33.297131 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.297094    5070 generic.go:356] "Generic (PLEG): container finished" podID="3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede" containerID="3667ff4c14a711e5b3816efb81e3455342d1d124d7093301d42af5b12378791e" exitCode=0
Nov 03 08:17:33.320976 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.320794    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abf1628-8c68-450c-a458-eefa78b239b3-istio-token" pod "2abf1628-8c68-450c-a458-eefa78b239b3" (UID: "2abf1628-8c68-450c-a458-eefa78b239b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:33.352591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.325953    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused" start-of-body=
Nov 03 08:17:33.352591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.326026    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk" podUID="212c33bc-85a6-4ff5-a96a-eda0064fab63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.202:15021/healthz/ready\": dial tcp 10.132.0.202:15021: connect: connection refused"
Nov 03 08:17:33.352591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.331494    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2abf1628-8c68-450c-a458-eefa78b239b3-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.352591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.331526    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.352591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.331552    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.352591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.331568    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/870ca6ae-460c-4c9e-9d51-5140a7025bc5-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.352591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.331587    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.352591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.331604    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0c52e6-b384-411f-9c3a-c2d76c149238-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.352591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.331626    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2abf1628-8c68-450c-a458-eefa78b239b3-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.352591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.331642    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.352591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.331657    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/181a2d26-3317-450e-97be-c0082d47a27a-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.352591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.348551    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2abf1628-8c68-450c-a458-eefa78b239b3-istio-podinfo" pod "2abf1628-8c68-450c-a458-eefa78b239b3" (UID: "2abf1628-8c68-450c-a458-eefa78b239b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:17:33.469717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.374519    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-istio-envoy" pod "2abf1628-8c68-450c-a458-eefa78b239b3" (UID: "2abf1628-8c68-450c-a458-eefa78b239b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:33.469717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.375196    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/870ca6ae-460c-4c9e-9d51-5140a7025bc5-kube-api-access-2rw5b" pod "870ca6ae-460c-4c9e-9d51-5140a7025bc5" (UID: "870ca6ae-460c-4c9e-9d51-5140a7025bc5"). InnerVolumeSpecName "kube-api-access-2rw5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:33.469717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.455947    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.469717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.455979    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2abf1628-8c68-450c-a458-eefa78b239b3-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.469717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.455996    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rw5b\" (UniqueName: \"kubernetes.io/projected/870ca6ae-460c-4c9e-9d51-5140a7025bc5-kube-api-access-2rw5b\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.469717 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:17:33.462008    5070 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.438s"
Nov 03 08:17:33.469717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.462064    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq"]
Nov 03 08:17:33.469717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.463422    5070 cpu_manager.go:414] "containerMap: removing container" podUID="870ca6ae-460c-4c9e-9d51-5140a7025bc5" containerName="istio-proxy"
Nov 03 08:17:33.469717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.463444    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="870ca6ae-460c-4c9e-9d51-5140a7025bc5" containerName="istio-proxy"
Nov 03 08:17:33.469717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.463732    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="870ca6ae-460c-4c9e-9d51-5140a7025bc5" containerName="istio-proxy"
Nov 03 08:17:33.476977 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.475455    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2abf1628-8c68-450c-a458-eefa78b239b3-kube-api-access-fhk94" pod "2abf1628-8c68-450c-a458-eefa78b239b3" (UID: "2abf1628-8c68-450c-a458-eefa78b239b3"). InnerVolumeSpecName "kube-api-access-fhk94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:33.579115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.538284    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-istio-data" pod "2abf1628-8c68-450c-a458-eefa78b239b3" (UID: "2abf1628-8c68-450c-a458-eefa78b239b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:33.579115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.539052    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpp2p\" (UniqueName: \"kubernetes.io/projected/84094812-6bd0-401a-a91c-d3d24e71501f-kube-api-access-kpp2p\") pod \"payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s\" (UID: \"84094812-6bd0-401a-a91c-d3d24e71501f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s"
Nov 03 08:17:33.579115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.557330    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s"
Nov 03 08:17:33.579115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.559017    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2abf1628-8c68-450c-a458-eefa78b239b3-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.579115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.559072    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhk94\" (UniqueName: \"kubernetes.io/projected/2abf1628-8c68-450c-a458-eefa78b239b3-kube-api-access-fhk94\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:33.658935 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:17:33.610045    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d65b5b6_6c50_43be_a0cd_6cd3a7f00ede.slice/crio-3667ff4c14a711e5b3816efb81e3455342d1d124d7093301d42af5b12378791e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cb7af3_82c2_482c_aa2d_510c50735024.slice/crio-2e46fc0ae212af5ae9cfd6003a9203d3f9bc74178aeb56ad9dbc7926011bbb5c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d65b5b6_6c50_43be_a0cd_6cd3a7f00ede.slice/crio-conmon-3667ff4c14a711e5b3816efb81e3455342d1d124d7093301d42af5b12378791e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abf1628_8c68_450c_a458_eefa78b239b3.slice/crio-d93bb40e960d77605b81abd3b9d587c894c99bcfd66e65d1133c662cfa9e242b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5211e023_fa82_4010_939e_89afebb9e6b8.slice/crio-conmon-a0f5d44418a8ed4fdedf0b5f3452626063bda534bdd4c631195fca4a944673aa.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181a2d26_3317_450e_97be_c0082d47a27a.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 08:17:33.748155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.673881    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-n28db_5738916f-8202-40ed-823e-90f7c9cc4ec9/csi-driver/0.log"
Nov 03 08:17:33.748155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.696983    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:17:33.748155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.697019    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65" event={"ID":"9ecd3f55-84e0-4321-8d19-900388b6a91f","Type":"ContainerStarted","Data":"56f0d09b0bdbf2484e21e231cd772e9c0c74137e6a17ed692f3833c07d7da44c"}
Nov 03 08:17:33.748155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.697052    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m" event={"ID":"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede","Type":"ContainerDied","Data":"3667ff4c14a711e5b3816efb81e3455342d1d124d7093301d42af5b12378791e"}
Nov 03 08:17:33.748155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.697105    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:17:33.748155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.697140    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:17:33.748155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.697223    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n28db" event={"ID":"5738916f-8202-40ed-823e-90f7c9cc4ec9","Type":"ContainerStarted","Data":"6b7ab4907fb4bf4d7232e3fc7a368b6b08f2f1bbced876cede6879ce724608b7"}
Nov 03 08:17:33.748155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.697272    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:17:33.748155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.697305    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:17:33.748155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.697337    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:17:33.748155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.697355    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq"]
Nov 03 08:17:33.748155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.697389    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g"]
Nov 03 08:17:33.748155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.745314    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq"
Nov 03 08:17:33.765592 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.765280    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:33.816406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.808338    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:17:33.816406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.808375    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g"]
Nov 03 08:17:33.816406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.808464    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:33.816406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.808499    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"]
Nov 03 08:17:33.816406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.808533    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9"]
Nov 03 08:17:33.825229 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.825096    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g"
Nov 03 08:17:33.827898 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.827360    5070 cpu_manager.go:414] "containerMap: removing container" podUID="3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede" containerName="istio-proxy"
Nov 03 08:17:33.827898 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.827382    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede" containerName="istio-proxy"
Nov 03 08:17:33.827898 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.827638    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede" containerName="istio-proxy"
Nov 03 08:17:33.837019 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.828952    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"
Nov 03 08:17:33.842266 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.840155    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"
Nov 03 08:17:33.842266 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.841477    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7f6f94\""
Nov 03 08:17:33.842266 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.841643    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7f6f94-dockercfg-gt68b\""
Nov 03 08:17:33.849478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.843334    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq7qf\" (UniqueName: \"kubernetes.io/projected/6b7840f4-acec-4021-9ab3-c15fc29f5f8b-kube-api-access-mq7qf\") pod \"payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq\" (UID: \"6b7840f4-acec-4021-9ab3-c15fc29f5f8b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq"
Nov 03 08:17:33.849478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.843570    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7840f4-acec-4021-9ab3-c15fc29f5f8b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq\" (UID: \"6b7840f4-acec-4021-9ab3-c15fc29f5f8b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq"
Nov 03 08:17:33.961868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.944866    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-kube-api-access-b95ll\" (UniqueName: \"kubernetes.io/projected/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-kube-api-access-b95ll\") pod \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") "
Nov 03 08:17:33.961868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.945045    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-podinfo\") pod \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") "
Nov 03 08:17:33.961868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.945167    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-workload-socket\") pod \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") "
Nov 03 08:17:33.961868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.945208    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-token\" (UniqueName: \"kubernetes.io/projected/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-token\") pod \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") "
Nov 03 08:17:33.961868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.945241    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-envoy\") pod \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") "
Nov 03 08:17:33.961868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.945330    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-data\") pod \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") "
Nov 03 08:17:33.961868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.945364    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-workload-certs\") pod \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") "
Nov 03 08:17:33.961868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.945395    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istiod-ca-cert\") pod \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") "
Nov 03 08:17:33.961868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.945580    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-credential-socket\") pod \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\" (UID: \"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede\") "
Nov 03 08:17:33.961868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.946066    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tnkx\" (UniqueName: \"kubernetes.io/projected/71db1c49-1b13-498f-a39f-267b76067aee-kube-api-access-6tnkx\") pod \"payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g\" (UID: \"71db1c49-1b13-498f-a39f-267b76067aee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g"
Nov 03 08:17:33.961868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.946154    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq7qf\" (UniqueName: \"kubernetes.io/projected/6b7840f4-acec-4021-9ab3-c15fc29f5f8b-kube-api-access-mq7qf\") pod \"payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq\" (UID: \"6b7840f4-acec-4021-9ab3-c15fc29f5f8b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq"
Nov 03 08:17:33.961868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.946225    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-data" pod "3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede" (UID: "3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:33.961868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.946750    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-workload-certs" pod "3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede" (UID: "3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:33.961868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.946794    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-workload-socket" pod "3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede" (UID: "3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:33.961868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.947054    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istiod-ca-cert" pod "3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede" (UID: "3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:17:33.961868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.947275    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-credential-socket" pod "3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede" (UID: "3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:34.074125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.963476    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7840f4-acec-4021-9ab3-c15fc29f5f8b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq\" (UID: \"6b7840f4-acec-4021-9ab3-c15fc29f5f8b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq"
Nov 03 08:17:34.074125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.963644    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71db1c49-1b13-498f-a39f-267b76067aee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g\" (UID: \"71db1c49-1b13-498f-a39f-267b76067aee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g"
Nov 03 08:17:34.074125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.963835    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:34.074125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.963857    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:34.074125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.963880    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:34.074125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.963895    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:34.074125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.963910    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:34.074125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.964875    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7840f4-acec-4021-9ab3-c15fc29f5f8b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq\" (UID: \"6b7840f4-acec-4021-9ab3-c15fc29f5f8b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq"
Nov 03 08:17:34.074125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.969840    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-envoy" pod "3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede" (UID: "3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:34.074125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.971161    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-token" pod "3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede" (UID: "3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:34.074125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.971480    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-podinfo" pod "3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede" (UID: "3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:17:34.074125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.973740    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-kube-api-access-b95ll" pod "3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede" (UID: "3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede"). InnerVolumeSpecName "kube-api-access-b95ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:34.074125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:33.984822    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq7qf\" (UniqueName: \"kubernetes.io/projected/6b7840f4-acec-4021-9ab3-c15fc29f5f8b-kube-api-access-mq7qf\") pod \"payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq\" (UID: \"6b7840f4-acec-4021-9ab3-c15fc29f5f8b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq"
Nov 03 08:17:34.074125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.050182    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq"
Nov 03 08:17:34.074125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.066088    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71db1c49-1b13-498f-a39f-267b76067aee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g\" (UID: \"71db1c49-1b13-498f-a39f-267b76067aee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g"
Nov 03 08:17:34.074125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.066273    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tnkx\" (UniqueName: \"kubernetes.io/projected/71db1c49-1b13-498f-a39f-267b76067aee-kube-api-access-6tnkx\") pod \"payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g\" (UID: \"71db1c49-1b13-498f-a39f-267b76067aee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g"
Nov 03 08:17:34.074125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.067406    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b95ll\" (UniqueName: \"kubernetes.io/projected/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-kube-api-access-b95ll\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:34.074125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.068969    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:34.074125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.069020    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:34.074125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.069143    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:34.074125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.070550    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71db1c49-1b13-498f-a39f-267b76067aee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g\" (UID: \"71db1c49-1b13-498f-a39f-267b76067aee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g"
Nov 03 08:17:34.100187 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.095883    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tnkx\" (UniqueName: \"kubernetes.io/projected/71db1c49-1b13-498f-a39f-267b76067aee-kube-api-access-6tnkx\") pod \"payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g\" (UID: \"71db1c49-1b13-498f-a39f-267b76067aee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g"
Nov 03 08:17:34.111250 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.109354    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9"]
Nov 03 08:17:34.111250 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.109440    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:17:34.111250 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.109468    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657" event={"ID":"870ca6ae-460c-4c9e-9d51-5140a7025bc5","Type":"ContainerDied","Data":"c3c6f2dd2b7e88a5894f4b2a43acec5171e401dd103ea30af7c3892578a491d6"}
Nov 03 08:17:34.111250 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.109505    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9"
Nov 03 08:17:34.111250 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.109541    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:17:34.111250 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.110173    5070 scope.go:122] "RemoveContainer" containerID="283a6618d7b74aa3c22c7e8daea17b88f1a06ebefb40f4b446c0be1e4beeabc2"
Nov 03 08:17:34.131041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.128674    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g"
Nov 03 08:17:34.190469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.173171    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m9n2\" (UniqueName: \"kubernetes.io/projected/115f98fe-906d-4427-ab88-ab890bd2ab66-kube-api-access-2m9n2\") pod \"payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9\" (UID: \"115f98fe-906d-4427-ab88-ab890bd2ab66\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9"
Nov 03 08:17:34.190469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.173689    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/115f98fe-906d-4427-ab88-ab890bd2ab66-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9\" (UID: \"115f98fe-906d-4427-ab88-ab890bd2ab66\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9"
Nov 03 08:17:34.241971 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.208551    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:17:34.241971 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.208651    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:17:34.241971 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.208677    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"]
Nov 03 08:17:34.275781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.275672    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m9n2\" (UniqueName: \"kubernetes.io/projected/115f98fe-906d-4427-ab88-ab890bd2ab66-kube-api-access-2m9n2\") pod \"payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9\" (UID: \"115f98fe-906d-4427-ab88-ab890bd2ab66\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9"
Nov 03 08:17:34.277512 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.277455    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/115f98fe-906d-4427-ab88-ab890bd2ab66-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9\" (UID: \"115f98fe-906d-4427-ab88-ab890bd2ab66\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9"
Nov 03 08:17:34.365641 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.362006    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/115f98fe-906d-4427-ab88-ab890bd2ab66-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9\" (UID: \"115f98fe-906d-4427-ab88-ab890bd2ab66\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9"
Nov 03 08:17:34.365641 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.362811    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"]
Nov 03 08:17:34.365641 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.362916    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:34.365641 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.362954    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"]
Nov 03 08:17:34.365641 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.363355    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.413808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.413137    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.413808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.413187    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qprgg\" (UniqueName: \"kubernetes.io/projected/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-kube-api-access-qprgg\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.413808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.413237    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.413808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.413265    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.413808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.413484    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.413808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.413630    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.413808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.413785    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.414317 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.414021    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.414317 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.414067    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.465903 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.465839    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m9n2\" (UniqueName: \"kubernetes.io/projected/115f98fe-906d-4427-ab88-ab890bd2ab66-kube-api-access-2m9n2\") pod \"payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9\" (UID: \"115f98fe-906d-4427-ab88-ab890bd2ab66\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9"
Nov 03 08:17:34.482676 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.481152    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"]
Nov 03 08:17:34.482676 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.481205    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"]
Nov 03 08:17:34.484588 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.484168    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.520788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.519942    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.520788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.520081    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.520788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.520168    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.520788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.520319    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.520788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.520344    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.520788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.520441    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.520788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.520472    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qprgg\" (UniqueName: \"kubernetes.io/projected/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-kube-api-access-qprgg\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.520788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.520499    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.520788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.520527    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.528156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.527277    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.528156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.527314    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.528156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.527582    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.528156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.527834    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.532959 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.528987    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.550398 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.549142    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.554686 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.554576    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qprgg\" (UniqueName: \"kubernetes.io/projected/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-kube-api-access-qprgg\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.567877 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.558412    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.567877 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.563000    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.574556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.573803    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"]
Nov 03 08:17:34.574556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.573907    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:17:34.574556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.573973    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"]
Nov 03 08:17:34.574823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.574684    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.637278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.636541    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.637278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.636590    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.637278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.636629    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f813310f-254a-4b89-940c-06d7b335a112-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.637278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.636657    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.637278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.636689    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.637278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.636842    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.637278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.636891    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.637278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.636920    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjxdm\" (UniqueName: \"kubernetes.io/projected/f813310f-254a-4b89-940c-06d7b335a112-kube-api-access-jjxdm\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.637278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.637019    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f813310f-254a-4b89-940c-06d7b335a112-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.637278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.637073    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.637278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.637119    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.637278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.637203    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.637278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.637244    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.739875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.637534    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.739875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.637615    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbhvx\" (UniqueName: \"kubernetes.io/projected/c3d60ea0-edd4-4c91-87be-9b414765dd7c-kube-api-access-zbhvx\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.739875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.637648    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f813310f-254a-4b89-940c-06d7b335a112-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.739875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.637672    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.739875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.637700    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.739875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.675182    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9"
Nov 03 08:17:34.739875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.699052    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"]
Nov 03 08:17:34.739875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.699110    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"]
Nov 03 08:17:34.739875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.699132    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-hq657"]
Nov 03 08:17:34.739875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.699161    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"]
Nov 03 08:17:34.739875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.699182    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-6869q"]
Nov 03 08:17:34.739875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.699307    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.778095    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.778147    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.778183    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.778247    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbhvx\" (UniqueName: \"kubernetes.io/projected/c3d60ea0-edd4-4c91-87be-9b414765dd7c-kube-api-access-zbhvx\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.778271    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.778304    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f813310f-254a-4b89-940c-06d7b335a112-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.778328    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.778358    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.778445    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.778497    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.778531    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.778561    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.778595    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f813310f-254a-4b89-940c-06d7b335a112-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.778626    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.778657    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.778699    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.778835    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.778888    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.780203    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjxdm\" (UniqueName: \"kubernetes.io/projected/f813310f-254a-4b89-940c-06d7b335a112-kube-api-access-jjxdm\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.780303    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f813310f-254a-4b89-940c-06d7b335a112-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.780363    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp6sp\" (UniqueName: \"kubernetes.io/projected/5edae0d0-8211-41ba-8b4b-154fa2b3f840-kube-api-access-bp6sp\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.780394    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.780429    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.780458    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.780505    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.780575    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.780642    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.836739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.828964    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.857563    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.882227    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.882278    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.882527    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp6sp\" (UniqueName: \"kubernetes.io/projected/5edae0d0-8211-41ba-8b4b-154fa2b3f840-kube-api-access-bp6sp\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.882550    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.882578    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.882667    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.882742    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.882856    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.883087    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.883853    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.883889    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm" podUID="5e519992-15af-4d72-ba82-190c90ab0aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.929055    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.929157    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.929340    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.929486    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s" event={"ID":"312a9c42-79df-4b2d-b430-0c02d7c37200","Type":"ContainerStarted","Data":"f9a91dc4463a8e8c4deb22688cea1686bf757ff9ef5c83b0d7980ed000b3b2af"}
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.929517    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.929542    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s" event={"ID":"312a9c42-79df-4b2d-b430-0c02d7c37200","Type":"ContainerStarted","Data":"1a0661c239177ea72b63fbd95b1f755bbdaad3e776d38c229d4bba50955fb381"}
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.929599    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f813310f-254a-4b89-940c-06d7b335a112-istio-podinfo\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.929624    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.929909    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-credential-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:34.954528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:34.929944    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:35.012851 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.012541    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-istio-data\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:35.012851 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.012807    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-workload-socket\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:35.013109 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.012931    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f813310f-254a-4b89-940c-06d7b335a112-istiod-ca-cert\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:35.013109 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.013006    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-workload-certs\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:35.013521 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.013459    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:35.013521 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.013465    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:35.013707 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.013532    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:35.036316 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.036280    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq"]
Nov 03 08:17:35.047132 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.047069    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:35.047252 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.047155    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-envoy\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:35.047724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.047443    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjxdm\" (UniqueName: \"kubernetes.io/projected/f813310f-254a-4b89-940c-06d7b335a112-kube-api-access-jjxdm\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:35.047724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.047679    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f813310f-254a-4b89-940c-06d7b335a112-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:35.051152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.051125    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-token\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:35.051230 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.051135    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:35.051230 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.051175    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp6sp\" (UniqueName: \"kubernetes.io/projected/5edae0d0-8211-41ba-8b4b-154fa2b3f840-kube-api-access-bp6sp\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:35.051525 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.051495    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:35.051752 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.051712    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbhvx\" (UniqueName: \"kubernetes.io/projected/c3d60ea0-edd4-4c91-87be-9b414765dd7c-kube-api-access-zbhvx\") pod \"e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") " pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:35.052000 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.051799    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:35.155322 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:17:35.155168    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7840f4_acec_4021_9ab3_c15fc29f5f8b.slice/crio-8c327a769b79d4d6f54895468952bdb53623289b57ff92a20e6d2e16ccdc2b4d WatchSource:0}: Error finding container 8c327a769b79d4d6f54895468952bdb53623289b57ff92a20e6d2e16ccdc2b4d: Status 404 returned error can't find the container with id 8c327a769b79d4d6f54895468952bdb53623289b57ff92a20e6d2e16ccdc2b4d
Nov 03 08:17:35.158192 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.157383    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:35.179286 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.177846    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s"]
Nov 03 08:17:35.184822 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.184800    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g"]
Nov 03 08:17:35.229513 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.229395    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:35.269826 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:17:35.269613    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71db1c49_1b13_498f_a39f_267b76067aee.slice/crio-2b34f8adab1a1b8850e34d4cb74765813034f178b834e37f60d714e7cb6c4c38 WatchSource:0}: Error finding container 2b34f8adab1a1b8850e34d4cb74765813034f178b834e37f60d714e7cb6c4c38: Status 404 returned error can't find the container with id 2b34f8adab1a1b8850e34d4cb74765813034f178b834e37f60d714e7cb6c4c38
Nov 03 08:17:35.366016 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:17:35.365982    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84094812_6bd0_401a_a91c_d3d24e71501f.slice/crio-d4bc1da003ceb62e3d0ba571c06fdf384cf6b1c3b93e89e9ea38f110c63bed84 WatchSource:0}: Error finding container d4bc1da003ceb62e3d0ba571c06fdf384cf6b1c3b93e89e9ea38f110c63bed84: Status 404 returned error can't find the container with id d4bc1da003ceb62e3d0ba571c06fdf384cf6b1c3b93e89e9ea38f110c63bed84
Nov 03 08:17:35.517123 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.516187    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9"]
Nov 03 08:17:35.588184 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.583149    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:35.588184 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.584915    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"
Nov 03 08:17:35.588184 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.585487    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m" event={"ID":"3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede","Type":"ContainerDied","Data":"aee61dbba7ee551a74c72f15af2acd7f09833ad87b1e2ae3ffb6e39daa165c91"}
Nov 03 08:17:35.588184 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.585559    5070 scope.go:122] "RemoveContainer" containerID="3667ff4c14a711e5b3816efb81e3455342d1d124d7093301d42af5b12378791e"
Nov 03 08:17:35.652861 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.652787    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s" podStartSLOduration=61.652751502 podStartE2EDuration="1m1.652751502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:35.612880208 +0000 UTC m=+1453.727344163" watchObservedRunningTime="2026-11-03 08:17:35.652751502 +0000 UTC m=+1453.767215434"
Nov 03 08:17:35.800882 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.800545    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:35.850199 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.850168    5070 generic.go:356] "Generic (PLEG): container finished" podID="5211e023-fa82-4010-939e-89afebb9e6b8" containerID="a0f5d44418a8ed4fdedf0b5f3452626063bda534bdd4c631195fca4a944673aa" exitCode=0
Nov 03 08:17:35.854464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.853204    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d9549b467-hqhvz" event={"ID":"5211e023-fa82-4010-939e-89afebb9e6b8","Type":"ContainerDied","Data":"a0f5d44418a8ed4fdedf0b5f3452626063bda534bdd4c631195fca4a944673aa"}
Nov 03 08:17:35.854464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.853302    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7d9549b467-hqhvz" event={"ID":"5211e023-fa82-4010-939e-89afebb9e6b8","Type":"ContainerStarted","Data":"fd85927c439a5928cd9b8509c53414fc8e81d4fa9f2fc383e4657afd8f5f3025"}
Nov 03 08:17:35.855108 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.854769    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 08:17:35.895805 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.891114    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"
Nov 03 08:17:35.895805 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.893952    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"
Nov 03 08:17:35.896123 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.896030    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"
Nov 03 08:17:35.898266 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.898233    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 08:17:35.901791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.901714    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dd9d888fc-qjx65"
Nov 03 08:17:35.930741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.930681    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"]
Nov 03 08:17:35.938445 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.938387    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-62d7m"]
Nov 03 08:17:35.978914 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.978880    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"]
Nov 03 08:17:35.984359 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.984327    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-qm9rs"]
Nov 03 08:17:35.992133 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.992028    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"]
Nov 03 08:17:35.999243 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:35.999194    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-l7v6r"]
Nov 03 08:17:36.010125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.010078    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"]
Nov 03 08:17:36.017067 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.017043    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-255z6"]
Nov 03 08:17:36.035895 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.035867    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="181a2d26-3317-450e-97be-c0082d47a27a" path="/var/lib/kubelet/pods/181a2d26-3317-450e-97be-c0082d47a27a/volumes"
Nov 03 08:17:36.152565 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.152174    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2abf1628-8c68-450c-a458-eefa78b239b3" path="/var/lib/kubelet/pods/2abf1628-8c68-450c-a458-eefa78b239b3/volumes"
Nov 03 08:17:36.305247 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.237626    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede" path="/var/lib/kubelet/pods/3d65b5b6-6c50-43be-a0cd-6cd3a7f00ede/volumes"
Nov 03 08:17:36.305247 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.255480    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f0c52e6-b384-411f-9c3a-c2d76c149238" path="/var/lib/kubelet/pods/7f0c52e6-b384-411f-9c3a-c2d76c149238/volumes"
Nov 03 08:17:36.305247 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.256209    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="870ca6ae-460c-4c9e-9d51-5140a7025bc5" path="/var/lib/kubelet/pods/870ca6ae-460c-4c9e-9d51-5140a7025bc5/volumes"
Nov 03 08:17:36.305247 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.257051    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1cb7af3-82c2-482c-aa2d-510c50735024" path="/var/lib/kubelet/pods/b1cb7af3-82c2-482c-aa2d-510c50735024/volumes"
Nov 03 08:17:36.305247 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.257561    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:36.305247 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.257674    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:36.422485 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.422226    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"]
Nov 03 08:17:36.456844 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.456546    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"]
Nov 03 08:17:36.588116 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.587502    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:36.588116 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.587606    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:36.588116 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.587654    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:36.758634 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.599170    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:36.758634 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.599258    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:36.758634 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.599305    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:36.802629 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:17:36.776035    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d60ea0_edd4_4c91_87be_9b414765dd7c.slice/crio-b4f1383a3f80bef9c2b8ad3445ee25eb447e5b333eb0591408058397a714229a WatchSource:0}: Error finding container b4f1383a3f80bef9c2b8ad3445ee25eb447e5b333eb0591408058397a714229a: Status 404 returned error can't find the container with id b4f1383a3f80bef9c2b8ad3445ee25eb447e5b333eb0591408058397a714229a
Nov 03 08:17:36.802629 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.778289    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"]
Nov 03 08:17:36.802629 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.779319    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:36.802629 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.779391    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:36.802629 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.779425    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:36.847969 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.833722    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"]
Nov 03 08:17:36.847969 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:17:36.838211    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf813310f_254a_4b89_940c_06d7b335a112.slice/crio-b4164b1f8cfa999d9d0292ef5b14aefce05b295d0fd3d8709df355e6c9973311 WatchSource:0}: Error finding container b4164b1f8cfa999d9d0292ef5b14aefce05b295d0fd3d8709df355e6c9973311: Status 404 returned error can't find the container with id b4164b1f8cfa999d9d0292ef5b14aefce05b295d0fd3d8709df355e6c9973311
Nov 03 08:17:36.847969 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.842320    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:36.847969 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.842404    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:36.847969 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:36.842454    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:37.107400 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:37.107373    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g" event={"ID":"71db1c49-1b13-498f-a39f-267b76067aee","Type":"ContainerStarted","Data":"2b34f8adab1a1b8850e34d4cb74765813034f178b834e37f60d714e7cb6c4c38"}
Nov 03 08:17:37.109302 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:37.109167    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd" event={"ID":"c3d60ea0-edd4-4c91-87be-9b414765dd7c","Type":"ContainerStarted","Data":"b4f1383a3f80bef9c2b8ad3445ee25eb447e5b333eb0591408058397a714229a"}
Nov 03 08:17:37.111652 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:37.111622    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9" event={"ID":"115f98fe-906d-4427-ab88-ab890bd2ab66","Type":"ContainerStarted","Data":"56f80d554748db89a158a1844f9ededc7dae37cd40bb8dfdf7bc7b86bbc0454c"}
Nov 03 08:17:37.113451 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:37.113426    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq" event={"ID":"6b7840f4-acec-4021-9ab3-c15fc29f5f8b","Type":"ContainerStarted","Data":"8c327a769b79d4d6f54895468952bdb53623289b57ff92a20e6d2e16ccdc2b4d"}
Nov 03 08:17:37.115333 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:37.114974    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s" event={"ID":"84094812-6bd0-401a-a91c-d3d24e71501f","Type":"ContainerStarted","Data":"d4bc1da003ceb62e3d0ba571c06fdf384cf6b1c3b93e89e9ea38f110c63bed84"}
Nov 03 08:17:37.118741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:37.117812    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8" event={"ID":"5edae0d0-8211-41ba-8b4b-154fa2b3f840","Type":"ContainerStarted","Data":"a5b91813b299ec77839285ac83314cf0609bc182255f99609dc020a06c874e14"}
Nov 03 08:17:37.124425 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:37.124398    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929" event={"ID":"f813310f-254a-4b89-940c-06d7b335a112","Type":"ContainerStarted","Data":"b4164b1f8cfa999d9d0292ef5b14aefce05b295d0fd3d8709df355e6c9973311"}
Nov 03 08:17:37.127241 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:37.127219    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm" event={"ID":"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d","Type":"ContainerStarted","Data":"0c17f4eebf184b37cad44175c7f8629da014f07ac29bb75c09cc9b64bd3cc916"}
Nov 03 08:17:37.127430 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:37.127410    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:37.129521 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:37.129283    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:17:37.888679 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:17:37.888636    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abf1628_8c68_450c_a458_eefa78b239b3.slice/crio-d93bb40e960d77605b81abd3b9d587c894c99bcfd66e65d1133c662cfa9e242b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cb7af3_82c2_482c_aa2d_510c50735024.slice/crio-2e46fc0ae212af5ae9cfd6003a9203d3f9bc74178aeb56ad9dbc7926011bbb5c.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.154360    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s" event={"ID":"84094812-6bd0-401a-a91c-d3d24e71501f","Type":"ContainerStarted","Data":"4df1c0060b45187901e2d6d0a4ef244479c0b5de76163af428e0c148684ba023"}
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.154460    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s"
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.157006    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8" event={"ID":"5edae0d0-8211-41ba-8b4b-154fa2b3f840","Type":"ContainerStarted","Data":"65447f53b80c7570e545e62a80ac6de929e011ddf6aca53a6715a7d0dbbcf33c"}
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.160981    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929" event={"ID":"f813310f-254a-4b89-940c-06d7b335a112","Type":"ContainerStarted","Data":"d67b2683a3325f8a6b25d5aacdac5136a40650214806444b57b9c707da90d03f"}
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.161542    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": read tcp 10.132.0.2:53042->10.132.0.176:15021: read: connection reset by peer" start-of-body=
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.161586    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": read tcp 10.132.0.2:53042->10.132.0.176:15021: read: connection reset by peer"
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.165336    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm" event={"ID":"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d","Type":"ContainerStarted","Data":"c0dac5751a2dd2612c2e5ab4921f8f6c0fe90caa76eeee71bb742ad563d7e24a"}
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.167787    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g" event={"ID":"71db1c49-1b13-498f-a39f-267b76067aee","Type":"ContainerStarted","Data":"ff17b2580aa1b6ef4db8b026cc4140f293f3eae6f9f660ab842e8a2c281adee2"}
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.170246    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd" event={"ID":"c3d60ea0-edd4-4c91-87be-9b414765dd7c","Type":"ContainerStarted","Data":"95c97a02c5b2b0d10a4ad46e0a3d7ad73a3e1a56ccb837840cf4f6983beea19e"}
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.172574    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9" event={"ID":"115f98fe-906d-4427-ab88-ab890bd2ab66","Type":"ContainerStarted","Data":"13c498fc2743821864b10fdd308ddfe1b1737fafcf5b82f4a65bfa5667c13279"}
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.172641    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9"
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.174985    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq" event={"ID":"6b7840f4-acec-4021-9ab3-c15fc29f5f8b","Type":"ContainerStarted","Data":"79aca1a1021c1bb1cf4fc500127cb274a1d12f6a454e58edf89f65f133fbe158"}
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.175046    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq"
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.179050    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s" podStartSLOduration=11.179032607 podStartE2EDuration="11.179032607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:38.178669112 +0000 UTC m=+1456.293133046" watchObservedRunningTime="2026-11-03 08:17:38.179032607 +0000 UTC m=+1456.293496541"
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.199300    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm" podStartSLOduration=7.199256497 podStartE2EDuration="7.199256497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:38.197525261 +0000 UTC m=+1456.311989204" watchObservedRunningTime="2026-11-03 08:17:38.199256497 +0000 UTC m=+1456.313720431"
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.217746    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq" podStartSLOduration=11.217729688 podStartE2EDuration="11.217729688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:38.216917269 +0000 UTC m=+1456.331381201" watchObservedRunningTime="2026-11-03 08:17:38.217729688 +0000 UTC m=+1456.332193620"
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.230654    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.234234    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.234302    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g" podStartSLOduration=9.234285094 podStartE2EDuration="9.234285094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:38.231252441 +0000 UTC m=+1456.345716375" watchObservedRunningTime="2026-11-03 08:17:38.234285094 +0000 UTC m=+1456.348749025"
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.234333    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd" podUID="c3d60ea0-edd4-4c91-87be-9b414765dd7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.261801    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd" podStartSLOduration=7.261782161 podStartE2EDuration="7.261782161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:38.25758989 +0000 UTC m=+1456.372053821" watchObservedRunningTime="2026-11-03 08:17:38.261782161 +0000 UTC m=+1456.376246089"
Nov 03 08:17:38.292358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.276478    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9" podStartSLOduration=9.276465863 podStartE2EDuration="9.276465863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:38.27405743 +0000 UTC m=+1456.388521360" watchObservedRunningTime="2026-11-03 08:17:38.276465863 +0000 UTC m=+1456.390929794"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.295229    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929" podStartSLOduration=7.2952130969999995 podStartE2EDuration="7.295213097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:38.292751707 +0000 UTC m=+1456.407215639" watchObservedRunningTime="2026-11-03 08:17:38.295213097 +0000 UTC m=+1456.409677033"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.317888    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8" podStartSLOduration=54.317870379 podStartE2EDuration="54.317870379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:38.313686343 +0000 UTC m=+1456.428150296" watchObservedRunningTime="2026-11-03 08:17:38.317870379 +0000 UTC m=+1456.432334312"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.564903    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.564960    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb" podUID="d915ac09-a22a-4e5d-943a-97cdeb2d9c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.587497    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.601282    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.601350    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929" podUID="f813310f-254a-4b89-940c-06d7b335a112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.830120    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.831838    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.831881    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm" podUID="0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.840128    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.841292    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:38.841329    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8" podUID="5edae0d0-8211-41ba-8b4b-154fa2b3f840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:39.278381    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:39.278443    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd" podUID="c3d60ea0-edd4-4c91-87be-9b414765dd7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:39.532724    5070 generic.go:356] "Generic (PLEG): container finished" podID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerID="9bd4a10841b8474e8baebcb447545c93c6cb1b79d710ac3e880d8227edf1f286" exitCode=0
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:39.533220    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" event={"ID":"acaf418a-affb-4714-a5ce-5673ed5be68e","Type":"ContainerDied","Data":"9bd4a10841b8474e8baebcb447545c93c6cb1b79d710ac3e880d8227edf1f286"}
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:39.545742    5070 scope.go:122] "RemoveContainer" containerID="9bd4a10841b8474e8baebcb447545c93c6cb1b79d710ac3e880d8227edf1f286"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:39.554599    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:39.776642    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:39.776714    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929" podUID="f813310f-254a-4b89-940c-06d7b335a112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:39.836068    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:39.836143    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm" podUID="0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:40.241588    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:40.241716    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:40.241788    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:40.278675    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:40.290400    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:40.290453    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8" podUID="5edae0d0-8211-41ba-8b4b-154fa2b3f840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:40.296206    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:40.345118    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:40.345171    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd" podUID="c3d60ea0-edd4-4c91-87be-9b414765dd7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:40.583182    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:40.585893    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:40.585943    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929" podUID="f813310f-254a-4b89-940c-06d7b335a112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:40.622881    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:40.763750    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:40.941784    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:40.941840    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm" podUID="0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:40.948796    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 03 08:17:40.958236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:40.948846    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8" podUID="5edae0d0-8211-41ba-8b4b-154fa2b3f840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 03 08:17:41.099793 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:41.096922    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:17:41.099793 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:41.097539    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:41.114440 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:41.114374    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:41.186813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:41.186145    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:41.249782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:41.243991    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 03 08:17:41.249782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:41.244058    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd" podUID="c3d60ea0-edd4-4c91-87be-9b414765dd7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 03 08:17:41.294799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:41.291897    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:17:41.294799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:41.292809    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:17:41.319783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:41.313880    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:41.319783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:41.314799    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:17:41.721254 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:41.721210    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:17:41.722686 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:41.722660    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:17:41.783850 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:41.734042    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:17:41.783850 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:41.734248    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:17:41.783850 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:41.754556    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 08:17:41.783850 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:41.754603    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929" podUID="f813310f-254a-4b89-940c-06d7b335a112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 08:17:41.783850 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:41.773341    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:41.783850 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:41.774120    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:17:42.154782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:42.146119    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:42.188782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:42.163022    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:17:42.188782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:42.187846    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:42.189062 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:42.189018    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:17:42.197752 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:42.197719    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 03 08:17:42.226013 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:42.225975    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm" podUID="0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 03 08:17:42.226565 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:42.197995    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:42.226736 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:42.197979    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 03 08:17:42.226905 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:42.226882    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8" podUID="5edae0d0-8211-41ba-8b4b-154fa2b3f840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 03 08:17:42.325784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:42.297842    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:17:42.553835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:42.547990    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 03 08:17:42.566884 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:42.566841    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd" podUID="c3d60ea0-edd4-4c91-87be-9b414765dd7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 03 08:17:42.709796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:42.611848    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:42.709796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:42.680082    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 08:17:42.709796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:42.709347    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929" podUID="f813310f-254a-4b89-940c-06d7b335a112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 08:17:42.735793 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:42.722736    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:17:42.780788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:42.780490    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:17:42.780788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:42.780568    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:17:42.841050 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:42.810927    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:42.888933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:42.873185    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:43.013981 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:43.013938    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz" podUID="33f324bd-79b1-4540-a090-7e4a9d92f10b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:43.016770 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:43.014487    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:17:43.397524 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:43.365561    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:43.449873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:43.449335    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 03 08:17:43.449873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:43.449415    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8" podUID="5edae0d0-8211-41ba-8b4b-154fa2b3f840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 03 08:17:43.465084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:43.461512    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:17:43.551813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:43.466202    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 03 08:17:43.551813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:43.466256    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm" podUID="0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 03 08:17:43.551813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:43.466351    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 03 08:17:43.551813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:43.466391    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd" podUID="c3d60ea0-edd4-4c91-87be-9b414765dd7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 03 08:17:43.638847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:43.554347    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:17:43.701920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:43.668116    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:43.977993 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:43.977886    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-gstjn" podUID="6960b432-0c5c-4c03-bcc5-dec9ff9466c7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:44.055151 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.017324    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:17:44.055151 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.035712    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:17:44.241871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.225386    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:17:44.241871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.225781    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:17:44.241871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.227961    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:17:44.262782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.251254    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 08:17:44.262782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.251316    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929" podUID="f813310f-254a-4b89-940c-06d7b335a112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 08:17:44.316585 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.316539    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 03 08:17:44.316934 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.316906    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd" podUID="c3d60ea0-edd4-4c91-87be-9b414765dd7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 03 08:17:44.322148 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.322120    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 03 08:17:44.322333 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.322310    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8" podUID="5edae0d0-8211-41ba-8b4b-154fa2b3f840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 03 08:17:44.358813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.352912    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:44.442162 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.442110    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 03 08:17:44.442808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.442748    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm" podUID="0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 03 08:17:44.471142 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.460711    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" event={"ID":"acaf418a-affb-4714-a5ce-5673ed5be68e","Type":"ContainerStarted","Data":"11a078b3ccf228e563e1e3b67d99f29a2327698ee3cc23f7dc65a574eccf3f83"}
Nov 03 08:17:44.471142 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.460878    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:17:44.570835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.480946    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:17:44.730467 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.730430    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 08:17:44.747511 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.747472    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929" podUID="f813310f-254a-4b89-940c-06d7b335a112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 08:17:44.788975 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.788946    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:17:44.794482 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.794444    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 03 08:17:44.794712 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.794687    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 03 08:17:44.964380 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.964342    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 03 08:17:44.964702 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.964670    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm" podUID="0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 03 08:17:44.979351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:44.979304    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 03 08:17:45.045956 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:45.045914    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8" podUID="5edae0d0-8211-41ba-8b4b-154fa2b3f840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 03 08:17:45.160780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:45.041340    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:17:45.264876 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:45.258932    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:17:45.279441 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:45.272893    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:45.279441 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:45.242928    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:17:45.279441 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:45.274275    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:45.279441 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:45.233690    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 03 08:17:45.279441 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:45.277689    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd" podUID="c3d60ea0-edd4-4c91-87be-9b414765dd7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 03 08:17:45.290878 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:45.290838    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:45.339628 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:45.339583    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:45.378245 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:45.360108    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:17:45.532635 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:45.521717    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:45.836847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:45.814208    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 03 08:17:45.836847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:45.814271    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 03 08:17:45.918724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:45.844106    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 03 08:17:45.934689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:45.883790    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 03 08:17:45.935035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:45.934984    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8" podUID="5edae0d0-8211-41ba-8b4b-154fa2b3f840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 03 08:17:45.935878 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:45.935165    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm" podUID="0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 03 08:17:45.936783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:45.889628    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 08:17:45.943478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:45.943426    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929" podUID="f813310f-254a-4b89-940c-06d7b335a112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 08:17:46.119815 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.108117    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:46.119815 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.108586    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:46.243094 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.234922    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 03 08:17:46.243094 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.234980    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd" podUID="c3d60ea0-edd4-4c91-87be-9b414765dd7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 03 08:17:46.298064 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.271626    5070 patch_prober.go:29] interesting pod/e2e-api-b-228695-openshift-default-547df686cf-n5bqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": read tcp 10.132.0.2:43758->10.132.0.175:15021: read: connection reset by peer" start-of-body=
Nov 03 08:17:46.298064 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.271677    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg" podUID="63d796bd-8897-4067-b1b4-e39aecd06b88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": read tcp 10.132.0.2:43758->10.132.0.175:15021: read: connection reset by peer"
Nov 03 08:17:46.298064 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.275601    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:46.298064 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.276488    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:17:46.298064 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.279109    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:17:46.324288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.298182    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:17:46.369647 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.368255    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:17:46.383040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.379058    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:46.528790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.528625    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:17:46.587346 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.587303    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 08:17:46.628152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.628105    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929" podUID="f813310f-254a-4b89-940c-06d7b335a112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 08:17:46.628803 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.610907    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:46.799845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.775291    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 03 08:17:46.799845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.775420    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 03 08:17:46.848785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.808464    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded"
Nov 03 08:17:46.892784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.859639    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body=
Nov 03 08:17:46.892784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.859698    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm" podUID="0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused"
Nov 03 08:17:46.892784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.860274    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 03 08:17:46.892784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:46.860313    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8" podUID="5edae0d0-8211-41ba-8b4b-154fa2b3f840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 03 08:17:46.892784 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:17:46.863000    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cb7af3_82c2_482c_aa2d_510c50735024.slice/crio-2e46fc0ae212af5ae9cfd6003a9203d3f9bc74178aeb56ad9dbc7926011bbb5c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abf1628_8c68_450c_a458_eefa78b239b3.slice/crio-d93bb40e960d77605b81abd3b9d587c894c99bcfd66e65d1133c662cfa9e242b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d796bd_8897_4067_b1b4_e39aecd06b88.slice/crio-b11717b193b9d6894c25850a626a8a1063989c6c412a8f4f6606a2e54788de64.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 08:17:47.179137 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:47.176161    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:47.272721 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:47.196437    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:47.272721 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:47.197197    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:17:47.272721 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:47.197259    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:17:47.348779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:47.323034    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:47.594166 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:47.379216    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:47.594166 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:47.411858    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:47.594166 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:47.444985    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:47.594166 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:47.446607    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:17:47.594166 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:47.446910    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:17:47.594166 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:47.534922    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:47.594166 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:47.538179    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 03 08:17:47.594166 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:47.538230    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd" podUID="c3d60ea0-edd4-4c91-87be-9b414765dd7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 03 08:17:47.887823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:47.882224    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body=
Nov 03 08:17:47.887823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:47.882292    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8" podUID="5edae0d0-8211-41ba-8b4b-154fa2b3f840" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused"
Nov 03 08:17:47.994123 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:47.976062    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 08:17:47.994123 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:47.976134    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929" podUID="f813310f-254a-4b89-940c-06d7b335a112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 08:17:47.994617 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:47.994587    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 03 08:17:47.994846 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:47.994810    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 03 08:17:48.181837 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:48.177447    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:48.208892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:48.200571    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:17:48.212262 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:48.210437    5070 generic.go:356] "Generic (PLEG): container finished" podID="63d796bd-8897-4067-b1b4-e39aecd06b88" containerID="b11717b193b9d6894c25850a626a8a1063989c6c412a8f4f6606a2e54788de64" exitCode=137
Nov 03 08:17:48.299872 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:48.299822    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body=
Nov 03 08:17:48.426707 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:48.426659    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd" podUID="c3d60ea0-edd4-4c91-87be-9b414765dd7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused"
Nov 03 08:17:48.464553 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:48.449532    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:48.655783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:48.587041    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:48.655783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:48.603971    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg" event={"ID":"63d796bd-8897-4067-b1b4-e39aecd06b88","Type":"ContainerDied","Data":"b11717b193b9d6894c25850a626a8a1063989c6c412a8f4f6606a2e54788de64"}
Nov 03 08:17:48.655783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:48.624396    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:17:48.657994 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:48.657950    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:48.680782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:48.660223    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:17:48.782388 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:48.702152    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:17:48.782895 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:48.702183    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:17:48.786820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:48.770207    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused" start-of-body=
Nov 03 08:17:48.787035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:48.786996    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929" podUID="f813310f-254a-4b89-940c-06d7b335a112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.211:15021/healthz/ready\": dial tcp 10.132.0.211:15021: connect: connection refused"
Nov 03 08:17:48.789361 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:48.789340    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 03 08:17:48.789530 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:48.789507    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 03 08:17:49.153731 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.153534    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:49.217615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.177532    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:17:49.217615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.198782    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:17:49.217615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.206702    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:17:49.217615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.206842    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:17:49.217615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.206918    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:17:49.217615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.207013    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:49.217615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.207084    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:17:49.217615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.207191    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:49.217615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.207260    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:17:49.217615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.207301    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:17:49.454363 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.448494    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:49.454363 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.448584    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:49.454363 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.448674    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:17:49.463773 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.463726    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-envoy\") pod \"63d796bd-8897-4067-b1b4-e39aecd06b88\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") "
Nov 03 08:17:49.464117 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.464096    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-data\") pod \"63d796bd-8897-4067-b1b4-e39aecd06b88\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") "
Nov 03 08:17:49.464326 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.464275    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-credential-socket\") pod \"63d796bd-8897-4067-b1b4-e39aecd06b88\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") "
Nov 03 08:17:49.465678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.464567    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-workload-certs\") pod \"63d796bd-8897-4067-b1b4-e39aecd06b88\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") "
Nov 03 08:17:49.466005 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.465986    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d796bd-8897-4067-b1b4-e39aecd06b88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63d796bd-8897-4067-b1b4-e39aecd06b88-istiod-ca-cert\") pod \"63d796bd-8897-4067-b1b4-e39aecd06b88\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") "
Nov 03 08:17:49.466255 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.466238    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-workload-socket\") pod \"63d796bd-8897-4067-b1b4-e39aecd06b88\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") "
Nov 03 08:17:49.466404 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.466385    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-token\" (UniqueName: \"kubernetes.io/projected/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-token\") pod \"63d796bd-8897-4067-b1b4-e39aecd06b88\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") "
Nov 03 08:17:49.466638 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.466619    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-podinfo\") pod \"63d796bd-8897-4067-b1b4-e39aecd06b88\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") "
Nov 03 08:17:49.489566 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.489462    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:17:49.536779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.535222    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-data" pod "63d796bd-8897-4067-b1b4-e39aecd06b88" (UID: "63d796bd-8897-4067-b1b4-e39aecd06b88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:49.536779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.535778    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-credential-socket" pod "63d796bd-8897-4067-b1b4-e39aecd06b88" (UID: "63d796bd-8897-4067-b1b4-e39aecd06b88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:49.536779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.536598    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d796bd-8897-4067-b1b4-e39aecd06b88-istiod-ca-cert" pod "63d796bd-8897-4067-b1b4-e39aecd06b88" (UID: "63d796bd-8897-4067-b1b4-e39aecd06b88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:17:49.537106 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.537022    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-workload-certs" pod "63d796bd-8897-4067-b1b4-e39aecd06b88" (UID: "63d796bd-8897-4067-b1b4-e39aecd06b88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:49.646783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.630996    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-workload-socket" pod "63d796bd-8897-4067-b1b4-e39aecd06b88" (UID: "63d796bd-8897-4067-b1b4-e39aecd06b88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:49.707843 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.701317    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:17:49.801792 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.797443    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:17:49.983500 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.817313    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d796bd-8897-4067-b1b4-e39aecd06b88-kube-api-access-9sfwl\" (UniqueName: \"kubernetes.io/projected/63d796bd-8897-4067-b1b4-e39aecd06b88-kube-api-access-9sfwl\") pod \"63d796bd-8897-4067-b1b4-e39aecd06b88\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") "
Nov 03 08:17:49.983500 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:49.951827    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:17:50.016783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.016463    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:17:50.066580 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.066532    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 03 08:17:50.078908 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.078862    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 03 08:17:50.122727 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.089324    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:50.170780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.165569    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:50.290785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.290128    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s"
Nov 03 08:17:50.310931 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.165256    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-envoy" pod "63d796bd-8897-4067-b1b4-e39aecd06b88" (UID: "63d796bd-8897-4067-b1b4-e39aecd06b88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:17:50.310931 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.178714    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:17:50.310931 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.301731    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm" podUID="5e519992-15af-4d72-ba82-190c90ab0aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:50.310931 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.230295    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-token" pod "63d796bd-8897-4067-b1b4-e39aecd06b88" (UID: "63d796bd-8897-4067-b1b4-e39aecd06b88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:50.310931 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.309450    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63d796bd-8897-4067-b1b4-e39aecd06b88-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:50.317842 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.311249    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:50.317842 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.311293    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:50.317842 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.311317    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:50.317842 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.311343    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:50.317842 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.312388    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": context deadline exceeded"
Nov 03 08:17:50.347830 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.346428    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-podinfo" pod "63d796bd-8897-4067-b1b4-e39aecd06b88" (UID: "63d796bd-8897-4067-b1b4-e39aecd06b88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:17:50.527862 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.509547    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:17:50.527862 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.516008    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:50.583919 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.557478    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:50.583919 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.557513    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:50.583919 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.557532    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63d796bd-8897-4067-b1b4-e39aecd06b88-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:50.832294 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.832195    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-qfg9j" podUID="67eb2350-8895-4e7d-b1f2-e78e1ff5cf23" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:51.037622 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.926703    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d796bd-8897-4067-b1b4-e39aecd06b88-kube-api-access-9sfwl" pod "63d796bd-8897-4067-b1b4-e39aecd06b88" (UID: "63d796bd-8897-4067-b1b4-e39aecd06b88"). InnerVolumeSpecName "kube-api-access-9sfwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:51.037622 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.761867    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:17:51.037622 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:50.992682    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:51.037622 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.025286    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d796bd-8897-4067-b1b4-e39aecd06b88-kube-api-access-9sfwl\" (UniqueName: \"kubernetes.io/projected/63d796bd-8897-4067-b1b4-e39aecd06b88-kube-api-access-9sfwl\") pod \"63d796bd-8897-4067-b1b4-e39aecd06b88\" (UID: \"63d796bd-8897-4067-b1b4-e39aecd06b88\") "
Nov 03 08:17:51.037622 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:17:51.026429    5070 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/63d796bd-8897-4067-b1b4-e39aecd06b88/volumes/kubernetes.io~projected/kube-api-access-9sfwl
Nov 03 08:17:51.037622 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.026504    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d796bd-8897-4067-b1b4-e39aecd06b88-kube-api-access-9sfwl" pod "63d796bd-8897-4067-b1b4-e39aecd06b88" (UID: "63d796bd-8897-4067-b1b4-e39aecd06b88"). InnerVolumeSpecName "kube-api-access-9sfwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:17:51.037622 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.032693    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:51.132018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.111404    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 03 08:17:51.132018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.111464    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 03 08:17:51.132018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.123836    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq"
Nov 03 08:17:51.132018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.123892    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:51.132018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.123939    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9"
Nov 03 08:17:51.132018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.123963    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml"]
Nov 03 08:17:51.132018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.124001    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:17:51.132018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.124044    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:51.132018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.124081    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d563d08-pqgml"]
Nov 03 08:17:51.132018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.124118    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:17:51.132018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.124157    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:17:51.132018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.124193    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"]
Nov 03 08:17:51.132018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.125404    5070 cpu_manager.go:414] "containerMap: removing container" podUID="63d796bd-8897-4067-b1b4-e39aecd06b88" containerName="istio-proxy"
Nov 03 08:17:51.132018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.125426    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d796bd-8897-4067-b1b4-e39aecd06b88" containerName="istio-proxy"
Nov 03 08:17:51.132018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.125629    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d796bd-8897-4067-b1b4-e39aecd06b88" containerName="istio-proxy"
Nov 03 08:17:51.135694 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.134384    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.170049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.150058    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:51.170049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.151438    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sfwl\" (UniqueName: \"kubernetes.io/projected/63d796bd-8897-4067-b1b4-e39aecd06b88-kube-api-access-9sfwl\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:17:51.170049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.153191    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"
Nov 03 08:17:51.170049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.156392    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"]
Nov 03 08:17:51.170049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.156593    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg" event={"ID":"63d796bd-8897-4067-b1b4-e39aecd06b88","Type":"ContainerDied","Data":"faae2c263262f291bd40eb3ba6bebb8859cb3d5d3536aaa53324d50f05071dae"}
Nov 03 08:17:51.170049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.156644    5070 scope.go:122] "RemoveContainer" containerID="b11717b193b9d6894c25850a626a8a1063989c6c412a8f4f6606a2e54788de64"
Nov 03 08:17:51.171985 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.170825    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:17:51.285267 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.256870    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.285267 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.257109    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.285267 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.257252    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.285267 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.257297    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.285267 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.257339    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.285267 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.257417    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.285267 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.257486    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.285267 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.257513    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tckst\" (UniqueName: \"kubernetes.io/projected/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-kube-api-access-tckst\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.285267 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.257552    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.295533 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.294510    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g"
Nov 03 08:17:51.304883 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:17:51.304834    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abf1628_8c68_450c_a458_eefa78b239b3.slice/crio-d93bb40e960d77605b81abd3b9d587c894c99bcfd66e65d1133c662cfa9e242b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e519992_15af_4d72_ba82_190c90ab0aa8.slice/crio-conmon-409bb03d7b86506848bbbbc07a8c1b45dc59431ec30d50f76ee16014d3da0711.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cb7af3_82c2_482c_aa2d_510c50735024.slice/crio-2e46fc0ae212af5ae9cfd6003a9203d3f9bc74178aeb56ad9dbc7926011bbb5c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d796bd_8897_4067_b1b4_e39aecd06b88.slice/crio-faae2c263262f291bd40eb3ba6bebb8859cb3d5d3536aaa53324d50f05071dae\": RecentStats: unable to find data in memory cache]"
Nov 03 08:17:51.305039 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:17:51.305005    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cb7af3_82c2_482c_aa2d_510c50735024.slice/crio-2e46fc0ae212af5ae9cfd6003a9203d3f9bc74178aeb56ad9dbc7926011bbb5c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abf1628_8c68_450c_a458_eefa78b239b3.slice/crio-d93bb40e960d77605b81abd3b9d587c894c99bcfd66e65d1133c662cfa9e242b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d796bd_8897_4067_b1b4_e39aecd06b88.slice/crio-faae2c263262f291bd40eb3ba6bebb8859cb3d5d3536aaa53324d50f05071dae\": RecentStats: unable to find data in memory cache]"
Nov 03 08:17:51.398864 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.396370    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.399224 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.399176    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.399422 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.399405    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.400776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.400734    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.401084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.401056    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.401611 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.401591    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.401843 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.401825    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tckst\" (UniqueName: \"kubernetes.io/projected/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-kube-api-access-tckst\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.402063 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.402043    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.402349 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.402332    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.445220 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.432986    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"]
Nov 03 08:17:51.445220 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.444261    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-workload-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.445220 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.444814    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-228695-openshift-default-547df686cf-n5bqg"]
Nov 03 08:17:51.448096 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.448044    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-credential-socket\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.448236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.448126    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-workload-certs\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.448552 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.448531    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-data\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.449831 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.449807    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istiod-ca-cert\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.498947 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.494362    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tckst\" (UniqueName: \"kubernetes.io/projected/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-kube-api-access-tckst\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.502597 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.500194    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-podinfo\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.543136 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.543086    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-token\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.543331 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.543294    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-envoy\") pod \"e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") " pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.635407 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.635148    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:51.789839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.782863    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 03 08:17:51.789839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:51.782941    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 03 08:17:52.116897 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:52.116857    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d3ca38f-b7bc-431d-a1f9-aa69afe33648" path="/var/lib/kubelet/pods/5d3ca38f-b7bc-431d-a1f9-aa69afe33648/volumes"
Nov 03 08:17:52.117792 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:52.117743    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d796bd-8897-4067-b1b4-e39aecd06b88" path="/var/lib/kubelet/pods/63d796bd-8897-4067-b1b4-e39aecd06b88/volumes"
Nov 03 08:17:52.409791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:52.409735    5070 generic.go:356] "Generic (PLEG): container finished" podID="5e519992-15af-4d72-ba82-190c90ab0aa8" containerID="409bb03d7b86506848bbbbc07a8c1b45dc59431ec30d50f76ee16014d3da0711" exitCode=0
Nov 03 08:17:52.419784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:52.410212    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm" event={"ID":"5e519992-15af-4d72-ba82-190c90ab0aa8","Type":"ContainerDied","Data":"409bb03d7b86506848bbbbc07a8c1b45dc59431ec30d50f76ee16014d3da0711"}
Nov 03 08:17:52.420671 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:52.420642    5070 scope.go:122] "RemoveContainer" containerID="409bb03d7b86506848bbbbc07a8c1b45dc59431ec30d50f76ee16014d3da0711"
Nov 03 08:17:52.460573 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:17:52.453046    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:17:52.551043 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:52.539541    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:52.551043 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:52.539640    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:52.551043 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:52.539693    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:52.781164 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:52.781131    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body=
Nov 03 08:17:52.781435 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:52.781409    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused"
Nov 03 08:17:52.840092 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:17:52.840054    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cb7af3_82c2_482c_aa2d_510c50735024.slice/crio-2e46fc0ae212af5ae9cfd6003a9203d3f9bc74178aeb56ad9dbc7926011bbb5c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abf1628_8c68_450c_a458_eefa78b239b3.slice/crio-d93bb40e960d77605b81abd3b9d587c894c99bcfd66e65d1133c662cfa9e242b.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 08:17:53.064448 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:53.064419    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:17:53.560895 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:53.560860    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:17:53.561290 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:53.560907    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb" podUID="d915ac09-a22a-4e5d-943a-97cdeb2d9c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:17:53.750269 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:17:53.750197    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb569224_4a9c_4da1_b5c4_cb6ea74b68bd.slice/crio-b0d027fcb178d8878e9d20641066ae94ec004a5866fd579a6f22a1c82646221b WatchSource:0}: Error finding container b0d027fcb178d8878e9d20641066ae94ec004a5866fd579a6f22a1c82646221b: Status 404 returned error can't find the container with id b0d027fcb178d8878e9d20641066ae94ec004a5866fd579a6f22a1c82646221b
Nov 03 08:17:53.750802 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:53.750781    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"]
Nov 03 08:17:53.753457 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:53.753419    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:53.753582 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:53.753502    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:53.753582 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:53.753550    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:17:53.780391 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:53.780365    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:17:53.781286 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:53.781268    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:17:54.513804 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:54.513677    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r" event={"ID":"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd","Type":"ContainerStarted","Data":"a967eb664dd1a02bc3f7caa6e51c62911ebadfe4e74a25d5e7a237a847450b90"}
Nov 03 08:17:54.513804 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:54.513734    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r" event={"ID":"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd","Type":"ContainerStarted","Data":"b0d027fcb178d8878e9d20641066ae94ec004a5866fd579a6f22a1c82646221b"}
Nov 03 08:17:54.516086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:54.516058    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm" event={"ID":"5e519992-15af-4d72-ba82-190c90ab0aa8","Type":"ContainerStarted","Data":"9c93261828483c8ee6e805de46f8a2dc36f1a94de5507e5444255362cf4b45f6"}
Nov 03 08:17:54.535944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:54.535905    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r" podStartSLOduration=70.535892045 podStartE2EDuration="1m10.535892045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:17:54.531107789 +0000 UTC m=+1472.645571720" watchObservedRunningTime="2026-11-03 08:17:54.535892045 +0000 UTC m=+1472.650355977"
Nov 03 08:17:54.556168 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:54.556142    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:17:54.556377 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:54.556349    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 03 08:17:54.556538 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:54.556404    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm" podUID="5e519992-15af-4d72-ba82-190c90ab0aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 03 08:17:54.636165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:54.636137    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:17:55.054785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:55.054296    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt"]
Nov 03 08:17:55.109158 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:55.087406    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37b13ea-dtglt"]
Nov 03 08:17:55.556310 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:55.556276    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 03 08:17:55.604373 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:55.556343    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm" podUID="5e519992-15af-4d72-ba82-190c90ab0aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 03 08:17:55.637148 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:55.637121    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 03 08:17:55.637563 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:55.637163    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r" podUID="bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 03 08:17:56.110864 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:56.104708    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f2321b-ee05-4de7-a27a-0e49d3102330" path="/var/lib/kubelet/pods/a5f2321b-ee05-4de7-a27a-0e49d3102330/volumes"
Nov 03 08:17:56.584840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:56.557460    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 03 08:17:56.584840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:56.557534    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm" podUID="5e519992-15af-4d72-ba82-190c90ab0aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 03 08:17:56.635800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:56.635753    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 03 08:17:56.635961 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:56.635817    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r" podUID="bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 03 08:17:57.148332 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:57.148290    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7d9549b467-hqhvz"
Nov 03 08:17:57.255662 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:17:57.255622    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cb7af3_82c2_482c_aa2d_510c50735024.slice/crio-2e46fc0ae212af5ae9cfd6003a9203d3f9bc74178aeb56ad9dbc7926011bbb5c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abf1628_8c68_450c_a458_eefa78b239b3.slice/crio-d93bb40e960d77605b81abd3b9d587c894c99bcfd66e65d1133c662cfa9e242b.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 08:17:57.565836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:57.557214    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 03 08:17:57.565836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:57.557269    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm" podUID="5e519992-15af-4d72-ba82-190c90ab0aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 03 08:17:57.636934 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:57.636576    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 03 08:17:57.636934 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:57.636637    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r" podUID="bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 03 08:17:58.560178 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:58.560137    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 03 08:17:58.560916 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:58.560878    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm" podUID="5e519992-15af-4d72-ba82-190c90ab0aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 03 08:17:58.645724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:58.645669    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 03 08:17:58.647497 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:58.647454    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r" podUID="bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 03 08:17:59.592889 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:59.592852    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 03 08:17:59.599320 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:59.593397    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm" podUID="5e519992-15af-4d72-ba82-190c90ab0aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 03 08:17:59.650837 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:59.635842    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 03 08:17:59.650837 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:17:59.635898    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r" podUID="bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 03 08:18:00.578603 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:00.578563    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 03 08:18:00.579034 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:00.578997    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm" podUID="5e519992-15af-4d72-ba82-190c90ab0aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 03 08:18:00.635863 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:00.635827    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 03 08:18:00.636424 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:00.636395    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r" podUID="bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 03 08:18:01.564015 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:01.563513    5070 patch_prober.go:29] interesting pod/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body=
Nov 03 08:18:01.564015 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:01.563572    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm" podUID="5e519992-15af-4d72-ba82-190c90ab0aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused"
Nov 03 08:18:01.638131 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:01.636241    5070 patch_prober.go:29] interesting pod/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused" start-of-body=
Nov 03 08:18:01.638131 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:01.636313    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r" podUID="bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.213:15021/healthz/ready\": dial tcp 10.132.0.213:15021: connect: connection refused"
Nov 03 08:18:02.663599 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:02.663564    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:18:02.664081 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:02.664020    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:18:02.692936 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:02.664834    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:18:02.692936 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:02.666289    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:18:02.692936 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:02.666706    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:18:02.692936 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:02.667471    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:18:06.983139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:06.983104    5070 generic.go:356] "Generic (PLEG): container finished" podID="d915ac09-a22a-4e5d-943a-97cdeb2d9c24" containerID="38969c7bdd9b825821042634a5aac777ff416473f595f646a7ca0043dac7cd4a" exitCode=0
Nov 03 08:18:06.983925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:06.983149    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb" event={"ID":"d915ac09-a22a-4e5d-943a-97cdeb2d9c24","Type":"ContainerDied","Data":"38969c7bdd9b825821042634a5aac777ff416473f595f646a7ca0043dac7cd4a"}
Nov 03 08:18:06.983925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:06.983749    5070 scope.go:122] "RemoveContainer" containerID="38969c7bdd9b825821042634a5aac777ff416473f595f646a7ca0043dac7cd4a"
Nov 03 08:18:06.990602 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:06.987877    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:18:06.990602 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:06.987966    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:18:06.990602 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:06.988010    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:18:07.513149 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:07.513110    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:18:07.558519 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:18:07.558474    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abf1628_8c68_450c_a458_eefa78b239b3.slice/crio-d93bb40e960d77605b81abd3b9d587c894c99bcfd66e65d1133c662cfa9e242b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cb7af3_82c2_482c_aa2d_510c50735024.slice/crio-2e46fc0ae212af5ae9cfd6003a9203d3f9bc74178aeb56ad9dbc7926011bbb5c.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 08:18:07.724054 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:18:07.724009    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cb7af3_82c2_482c_aa2d_510c50735024.slice/crio-2e46fc0ae212af5ae9cfd6003a9203d3f9bc74178aeb56ad9dbc7926011bbb5c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abf1628_8c68_450c_a458_eefa78b239b3.slice/crio-d93bb40e960d77605b81abd3b9d587c894c99bcfd66e65d1133c662cfa9e242b.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 08:18:07.995064 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:07.995022    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb" event={"ID":"d915ac09-a22a-4e5d-943a-97cdeb2d9c24","Type":"ContainerStarted","Data":"e07f6abee64d466bfa79ecc820f3501f1b879c5f0b289677d99f154f78040f1a"}
Nov 03 08:18:08.169222 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:08.169190    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:18:08.178992 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:08.178955    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 03 08:18:08.198786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:08.180788    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb" podUID="d915ac09-a22a-4e5d-943a-97cdeb2d9c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 03 08:18:09.180784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:09.178081    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 03 08:18:09.180784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:09.178136    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb" podUID="d915ac09-a22a-4e5d-943a-97cdeb2d9c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 03 08:18:10.170785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:10.166053    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 03 08:18:10.170785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:10.166121    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb" podUID="d915ac09-a22a-4e5d-943a-97cdeb2d9c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 03 08:18:11.163790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:11.162082    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 03 08:18:11.163790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:11.162139    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb" podUID="d915ac09-a22a-4e5d-943a-97cdeb2d9c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 03 08:18:12.180786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:12.178121    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 03 08:18:12.180786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:12.178176    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb" podUID="d915ac09-a22a-4e5d-943a-97cdeb2d9c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 03 08:18:13.170190 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:13.170148    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body=
Nov 03 08:18:13.171099 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:13.170376    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb" podUID="d915ac09-a22a-4e5d-943a-97cdeb2d9c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused"
Nov 03 08:18:13.586324 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:13.586246    5070 scope.go:122] "RemoveContainer" containerID="122c7c5ef4ec7bc47b6873ca187ce4168aac0e66b1ced13cb56eb2b38d3ca4a0"
Nov 03 08:18:13.599794 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:13.599732    5070 scope.go:122] "RemoveContainer" containerID="e719b3e626dc6538cd9eca46d8bad4fa011b80318345a2b254b98549ea65baf1"
Nov 03 08:18:13.612176 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:13.612159    5070 scope.go:122] "RemoveContainer" containerID="c96a23ab691a87d470784c9aa049512543d087810e8fc7695666836b49509f61"
Nov 03 08:18:13.622768 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:13.622737    5070 scope.go:122] "RemoveContainer" containerID="dc727e7e98197d4e51c0475c55df1de39f5abe6b7aefb14e94add299aa00faea"
Nov 03 08:18:13.635685 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:13.635665    5070 scope.go:122] "RemoveContainer" containerID="3e2844dbf69aedc60417036f71a0d061759f0d3b1a3d06307336a314c37bf73e"
Nov 03 08:18:13.656259 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:13.656235    5070 scope.go:122] "RemoveContainer" containerID="d93bb40e960d77605b81abd3b9d587c894c99bcfd66e65d1133c662cfa9e242b"
Nov 03 08:18:13.683242 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:13.683201    5070 scope.go:122] "RemoveContainer" containerID="2e46fc0ae212af5ae9cfd6003a9203d3f9bc74178aeb56ad9dbc7926011bbb5c"
Nov 03 08:18:14.350515 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:14.350479    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m"]
Nov 03 08:18:14.424868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:14.386255    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-950e06-hxf9m"]
Nov 03 08:18:14.496137 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:14.479079    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:18:14.496137 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:14.480936    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:18:14.496137 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:14.496016    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:18:16.028724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:16.028693    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30f04061-9793-4d5f-8189-141d7a74d515" path="/var/lib/kubelet/pods/30f04061-9793-4d5f-8189-141d7a74d515/volumes"
Nov 03 08:18:17.774348 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:18:17.774311    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cb7af3_82c2_482c_aa2d_510c50735024.slice/crio-2e46fc0ae212af5ae9cfd6003a9203d3f9bc74178aeb56ad9dbc7926011bbb5c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2abf1628_8c68_450c_a458_eefa78b239b3.slice/crio-d93bb40e960d77605b81abd3b9d587c894c99bcfd66e65d1133c662cfa9e242b.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 08:18:22.072453 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.072423    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm"]
Nov 03 08:18:22.084450 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.084416    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm"
Nov 03 08:18:22.084605 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.084505    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm"]
Nov 03 08:18:22.100468 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.100439    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:18:22.100625 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.100474    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:18:22.101084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.101064    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:18:22.121982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.121731    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:18:22.134216 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:18:22.134182    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:18:22.226224 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.226190    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dadc8621-5a99-4c99-a41d-a10cd5a73440-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-99410b-8p4nm\" (UID: \"dadc8621-5a99-4c99-a41d-a10cd5a73440\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm"
Nov 03 08:18:22.226463 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.226442    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqbgd\" (UniqueName: \"kubernetes.io/projected/dadc8621-5a99-4c99-a41d-a10cd5a73440-kube-api-access-vqbgd\") pod \"maas-api-revoke-keys-e2e-api-b-99410b-8p4nm\" (UID: \"dadc8621-5a99-4c99-a41d-a10cd5a73440\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm"
Nov 03 08:18:22.226653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.226633    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dadc8621-5a99-4c99-a41d-a10cd5a73440-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-99410b-8p4nm\" (UID: \"dadc8621-5a99-4c99-a41d-a10cd5a73440\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm"
Nov 03 08:18:22.328854 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.328744    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dadc8621-5a99-4c99-a41d-a10cd5a73440-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-99410b-8p4nm\" (UID: \"dadc8621-5a99-4c99-a41d-a10cd5a73440\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm"
Nov 03 08:18:22.329012 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.328920    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dadc8621-5a99-4c99-a41d-a10cd5a73440-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-99410b-8p4nm\" (UID: \"dadc8621-5a99-4c99-a41d-a10cd5a73440\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm"
Nov 03 08:18:22.329178 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.329159    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqbgd\" (UniqueName: \"kubernetes.io/projected/dadc8621-5a99-4c99-a41d-a10cd5a73440-kube-api-access-vqbgd\") pod \"maas-api-revoke-keys-e2e-api-b-99410b-8p4nm\" (UID: \"dadc8621-5a99-4c99-a41d-a10cd5a73440\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm"
Nov 03 08:18:22.329542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.329524    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dadc8621-5a99-4c99-a41d-a10cd5a73440-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-99410b-8p4nm\" (UID: \"dadc8621-5a99-4c99-a41d-a10cd5a73440\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm"
Nov 03 08:18:22.334897 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.334874    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dadc8621-5a99-4c99-a41d-a10cd5a73440-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-99410b-8p4nm\" (UID: \"dadc8621-5a99-4c99-a41d-a10cd5a73440\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm"
Nov 03 08:18:22.339382 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.339359    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqbgd\" (UniqueName: \"kubernetes.io/projected/dadc8621-5a99-4c99-a41d-a10cd5a73440-kube-api-access-vqbgd\") pod \"maas-api-revoke-keys-e2e-api-b-99410b-8p4nm\" (UID: \"dadc8621-5a99-4c99-a41d-a10cd5a73440\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm"
Nov 03 08:18:22.408112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.408088    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm"
Nov 03 08:18:22.409317 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.409300    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/3.log"
Nov 03 08:18:22.409317 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.409310    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/3.log"
Nov 03 08:18:22.425119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.425100    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-n28db_5738916f-8202-40ed-823e-90f7c9cc4ec9/csi-driver/0.log"
Nov 03 08:18:22.425849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.425116    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-n28db_5738916f-8202-40ed-823e-90f7c9cc4ec9/csi-driver/0.log"
Nov 03 08:18:22.805541 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:18:22.805496    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddadc8621_5a99_4c99_a41d_a10cd5a73440.slice/crio-09f21e67d4872d4bc5f67a96823172cb3cbbf8337ff01aede1736dae6bf2ff6b WatchSource:0}: Error finding container 09f21e67d4872d4bc5f67a96823172cb3cbbf8337ff01aede1736dae6bf2ff6b: Status 404 returned error can't find the container with id 09f21e67d4872d4bc5f67a96823172cb3cbbf8337ff01aede1736dae6bf2ff6b
Nov 03 08:18:22.805735 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:22.805722    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm"]
Nov 03 08:18:23.590291 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:23.590245    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm" event={"ID":"dadc8621-5a99-4c99-a41d-a10cd5a73440","Type":"ContainerStarted","Data":"72f8580ef1bf93779927eefa63c21d948f333a1c29ffa692d2f49778d54e5e16"}
Nov 03 08:18:23.590740 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:23.590351    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm" event={"ID":"dadc8621-5a99-4c99-a41d-a10cd5a73440","Type":"ContainerStarted","Data":"09f21e67d4872d4bc5f67a96823172cb3cbbf8337ff01aede1736dae6bf2ff6b"}
Nov 03 08:18:23.613883 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:23.613828    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm" podStartSLOduration=1.613812652 podStartE2EDuration="1.613812652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:18:23.606331253 +0000 UTC m=+1501.720795185" watchObservedRunningTime="2026-11-03 08:18:23.613812652 +0000 UTC m=+1501.728276583"
Nov 03 08:18:24.601430 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:24.601394    5070 generic.go:356] "Generic (PLEG): container finished" podID="dadc8621-5a99-4c99-a41d-a10cd5a73440" containerID="72f8580ef1bf93779927eefa63c21d948f333a1c29ffa692d2f49778d54e5e16" exitCode=0
Nov 03 08:18:24.601982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:24.601435    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm" event={"ID":"dadc8621-5a99-4c99-a41d-a10cd5a73440","Type":"ContainerDied","Data":"72f8580ef1bf93779927eefa63c21d948f333a1c29ffa692d2f49778d54e5e16"}
Nov 03 08:18:25.908373 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:25.908352    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm"
Nov 03 08:18:26.076097 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:26.076066    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dadc8621-5a99-4c99-a41d-a10cd5a73440-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dadc8621-5a99-4c99-a41d-a10cd5a73440-maas-api-xks-ca\") pod \"dadc8621-5a99-4c99-a41d-a10cd5a73440\" (UID: \"dadc8621-5a99-4c99-a41d-a10cd5a73440\") "
Nov 03 08:18:26.076965 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:26.076184    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dadc8621-5a99-4c99-a41d-a10cd5a73440-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dadc8621-5a99-4c99-a41d-a10cd5a73440-maas-api-openshift-ca\") pod \"dadc8621-5a99-4c99-a41d-a10cd5a73440\" (UID: \"dadc8621-5a99-4c99-a41d-a10cd5a73440\") "
Nov 03 08:18:26.076965 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:26.076238    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dadc8621-5a99-4c99-a41d-a10cd5a73440-kube-api-access-vqbgd\" (UniqueName: \"kubernetes.io/projected/dadc8621-5a99-4c99-a41d-a10cd5a73440-kube-api-access-vqbgd\") pod \"dadc8621-5a99-4c99-a41d-a10cd5a73440\" (UID: \"dadc8621-5a99-4c99-a41d-a10cd5a73440\") "
Nov 03 08:18:26.076965 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:26.076499    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dadc8621-5a99-4c99-a41d-a10cd5a73440-maas-api-openshift-ca" pod "dadc8621-5a99-4c99-a41d-a10cd5a73440" (UID: "dadc8621-5a99-4c99-a41d-a10cd5a73440"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:18:26.076965 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:26.076719    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dadc8621-5a99-4c99-a41d-a10cd5a73440-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:18:26.079235 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:26.079210    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dadc8621-5a99-4c99-a41d-a10cd5a73440-maas-api-xks-ca" pod "dadc8621-5a99-4c99-a41d-a10cd5a73440" (UID: "dadc8621-5a99-4c99-a41d-a10cd5a73440"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:18:26.079348 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:26.079319    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dadc8621-5a99-4c99-a41d-a10cd5a73440-kube-api-access-vqbgd" pod "dadc8621-5a99-4c99-a41d-a10cd5a73440" (UID: "dadc8621-5a99-4c99-a41d-a10cd5a73440"). InnerVolumeSpecName "kube-api-access-vqbgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:18:26.177961 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:26.177901    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqbgd\" (UniqueName: \"kubernetes.io/projected/dadc8621-5a99-4c99-a41d-a10cd5a73440-kube-api-access-vqbgd\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:18:26.177961 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:26.177922    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dadc8621-5a99-4c99-a41d-a10cd5a73440-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:18:26.620820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:26.620791    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm"
Nov 03 08:18:26.620985 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:26.620789    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm" event={"ID":"dadc8621-5a99-4c99-a41d-a10cd5a73440","Type":"ContainerDied","Data":"09f21e67d4872d4bc5f67a96823172cb3cbbf8337ff01aede1736dae6bf2ff6b"}
Nov 03 08:18:26.620985 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:26.620921    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09f21e67d4872d4bc5f67a96823172cb3cbbf8337ff01aede1736dae6bf2ff6b"
Nov 03 08:18:41.301972 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:41.301921    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:41.303195 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:41.303155    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:41.383913 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:41.383852    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:41.486552 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:41.486500    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:18:41.554842 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:41.528379    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:41.731784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:41.729885    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:42.176933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:42.176884    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:42.200971 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:42.200924    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:42.207775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:42.207698    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:42.251895 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:42.251853    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:42.302906 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:42.302858    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:42.304022 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:42.303995    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:18:42.332940 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:42.332897    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:42.333723 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:42.333695    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:18:42.389954 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:42.386855    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:42.389954 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:42.387443    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:18:42.503968 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:42.503863    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:18:42.531295 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:42.531261    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:18:43.153855 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:42.613919    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:18:43.153855 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:42.666412    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:18:43.153855 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:42.734855    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:43.153855 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:42.735428    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:18:43.208792 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:43.208674    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:43.271913 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:43.254263    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:18:43.271913 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:43.255436    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:18:43.271913 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:43.255497    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:18:43.332597 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:43.332550    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:43.333635 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:43.333608    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:18:43.378786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:43.352227    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:18:43.609912 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:43.376435    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:18:43.670001 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:43.392641    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:43.721782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:43.527272    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:43.721782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:43.640223    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:43.795802 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:43.746912    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:44.128964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:44.125274    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded"
Nov 03 08:18:44.167791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:44.166906    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:44.231783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:44.227018    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:44.258083 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:44.255357    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:18:44.371184 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:44.336890    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:44.545840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:44.530711    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:18:44.672805 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:44.667902    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:18:44.689897 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:44.689849    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:44.718318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:44.717247    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:44.723221 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:44.719713    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded"
Nov 03 08:18:44.742046 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:44.742017    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:18:44.759881 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:44.759844    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:45.168260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:45.168227    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:18:45.228516 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:45.228172    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:45.237721 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:45.237688    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:18:45.239082 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:45.239050    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t"
Nov 03 08:18:45.279782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:45.273010    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:18:45.286054 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:45.286030    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:18:45.307777 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:45.292513    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:18:45.307777 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:45.299370    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:18:45.307777 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:45.302407    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:18:52.321789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:52.131294    5070 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:18:52.372413 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:52.336597    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k" podUID="ae54e276-2450-4cef-98d9-e6cc85ed028a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:52.656863 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:18:52.656785    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:18:53.178342 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:53.177952    5070 patch_prober.go:29] interesting pod/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:18:53.178342 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:53.178019    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s" podUID="312a9c42-79df-4b2d-b430-0c02d7c37200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:18:58.440643 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:58.440609    5070 generic.go:356] "Generic (PLEG): container finished" podID="ae54e276-2450-4cef-98d9-e6cc85ed028a" containerID="b352c8ba7f3f94e3612f9c06f03b6efbd0508e20d4b388ee2bd4fddbfce28315" exitCode=0
Nov 03 08:18:58.441171 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:58.440702    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k" event={"ID":"ae54e276-2450-4cef-98d9-e6cc85ed028a","Type":"ContainerDied","Data":"b352c8ba7f3f94e3612f9c06f03b6efbd0508e20d4b388ee2bd4fddbfce28315"}
Nov 03 08:18:58.450041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:58.450017    5070 scope.go:122] "RemoveContainer" containerID="b352c8ba7f3f94e3612f9c06f03b6efbd0508e20d4b388ee2bd4fddbfce28315"
Nov 03 08:18:58.470809 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:58.470730    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:18:58.470956 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:58.470820    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:18:58.470956 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:58.470874    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:18:59.454210 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:18:59.454175    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k" event={"ID":"ae54e276-2450-4cef-98d9-e6cc85ed028a","Type":"ContainerStarted","Data":"75c299d56fbbae5553bf5fddbbc344e88842a50747c900fedbfd8c3af1d90bdc"}
Nov 03 08:19:00.395241 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:00.395135    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:19:00.395560 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:00.395524    5070 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 03 08:19:00.395863 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:00.395595    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k" podUID="ae54e276-2450-4cef-98d9-e6cc85ed028a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 03 08:19:01.382786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:01.382376    5070 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 03 08:19:01.382786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:01.382489    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k" podUID="ae54e276-2450-4cef-98d9-e6cc85ed028a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 03 08:19:02.409484 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:02.409442    5070 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 03 08:19:02.410948 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:02.409509    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k" podUID="ae54e276-2450-4cef-98d9-e6cc85ed028a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 03 08:19:03.383100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:03.383066    5070 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 03 08:19:03.383450 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:03.383411    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k" podUID="ae54e276-2450-4cef-98d9-e6cc85ed028a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 03 08:19:03.525910 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:03.525832    5070 generic.go:356] "Generic (PLEG): container finished" podID="312a9c42-79df-4b2d-b430-0c02d7c37200" containerID="f9a91dc4463a8e8c4deb22688cea1686bf757ff9ef5c83b0d7980ed000b3b2af" exitCode=0
Nov 03 08:19:03.526393 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:03.525902    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s" event={"ID":"312a9c42-79df-4b2d-b430-0c02d7c37200","Type":"ContainerDied","Data":"f9a91dc4463a8e8c4deb22688cea1686bf757ff9ef5c83b0d7980ed000b3b2af"}
Nov 03 08:19:03.526393 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:03.526386    5070 scope.go:122] "RemoveContainer" containerID="f9a91dc4463a8e8c4deb22688cea1686bf757ff9ef5c83b0d7980ed000b3b2af"
Nov 03 08:19:03.528553 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:03.528517    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:19:03.529880 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:03.528589    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:19:03.529880 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:03.528627    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:19:04.382871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:04.382839    5070 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 03 08:19:04.384700 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:04.382894    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k" podUID="ae54e276-2450-4cef-98d9-e6cc85ed028a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 03 08:19:04.536613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:04.536578    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s" event={"ID":"312a9c42-79df-4b2d-b430-0c02d7c37200","Type":"ContainerStarted","Data":"446b1efcb31edfa74e756f8ff3af675dd6ecbf93e4b559de3376f928b7a2f2d8"}
Nov 03 08:19:05.212942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:05.212907    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:19:05.214021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:05.213996    5070 patch_prober.go:29] interesting pod/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 03 08:19:05.214147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:05.214045    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s" podUID="312a9c42-79df-4b2d-b430-0c02d7c37200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 03 08:19:05.386781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:05.383105    5070 patch_prober.go:29] interesting pod/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused" start-of-body=
Nov 03 08:19:05.386781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:05.383167    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k" podUID="ae54e276-2450-4cef-98d9-e6cc85ed028a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.192:15021/healthz/ready\": dial tcp 10.132.0.192:15021: connect: connection refused"
Nov 03 08:19:05.816516 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:05.816483    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:19:06.212782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:06.212130    5070 patch_prober.go:29] interesting pod/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 03 08:19:06.212782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:06.212195    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s" podUID="312a9c42-79df-4b2d-b430-0c02d7c37200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 03 08:19:06.387375 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:06.387345    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:19:06.554820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:06.554707    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:19:07.133777 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:07.127899    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:19:07.218134 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:07.218096    5070 patch_prober.go:29] interesting pod/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 03 08:19:07.218531 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:07.218500    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s" podUID="312a9c42-79df-4b2d-b430-0c02d7c37200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 03 08:19:07.476840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:07.465820    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm"]
Nov 03 08:19:07.522779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:07.513168    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-99410b-8p4nm"]
Nov 03 08:19:08.028208 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:08.028171    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dadc8621-5a99-4c99-a41d-a10cd5a73440" path="/var/lib/kubelet/pods/dadc8621-5a99-4c99-a41d-a10cd5a73440/volumes"
Nov 03 08:19:08.211965 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:08.211930    5070 patch_prober.go:29] interesting pod/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 03 08:19:08.212790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:08.211979    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s" podUID="312a9c42-79df-4b2d-b430-0c02d7c37200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 03 08:19:09.213098 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:09.213064    5070 patch_prober.go:29] interesting pod/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 03 08:19:09.213493 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:09.213119    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s" podUID="312a9c42-79df-4b2d-b430-0c02d7c37200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 03 08:19:10.212808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:10.212771    5070 patch_prober.go:29] interesting pod/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body=
Nov 03 08:19:10.213004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:10.212845    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s" podUID="312a9c42-79df-4b2d-b430-0c02d7c37200" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused"
Nov 03 08:19:11.227884 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:11.227853    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:19:11.229470 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:11.229448    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:19:13.797110 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:13.797075    5070 scope.go:122] "RemoveContainer" containerID="fc07b5508d29a7ef97c22b7ee29b93d8468994652e554ac321024055707cfa3c"
Nov 03 08:19:13.815634 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:13.815610    5070 scope.go:122] "RemoveContainer" containerID="53a2cba5156777a60e8630d41bc3616a2ff09c9228839982721322597cc4768b"
Nov 03 08:19:13.828251 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:13.827916    5070 scope.go:122] "RemoveContainer" containerID="10755bb0f367772a9d9d8052a1c073ea3fc121f79ca584fb07b84da2be465b8a"
Nov 03 08:19:14.628341 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:14.628306    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"]
Nov 03 08:19:14.629322 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:14.629249    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k" podUID="ae54e276-2450-4cef-98d9-e6cc85ed028a" containerName="istio-proxy" containerID="cri-o://75c299d56fbbae5553bf5fddbbc344e88842a50747c900fedbfd8c3af1d90bdc" gracePeriod=30
Nov 03 08:19:19.787563 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:19.787532    5070 generic.go:356] "Generic (PLEG): container finished" podID="ae54e276-2450-4cef-98d9-e6cc85ed028a" containerID="75c299d56fbbae5553bf5fddbbc344e88842a50747c900fedbfd8c3af1d90bdc" exitCode=0
Nov 03 08:19:19.787983 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:19.787626    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k" event={"ID":"ae54e276-2450-4cef-98d9-e6cc85ed028a","Type":"ContainerDied","Data":"75c299d56fbbae5553bf5fddbbc344e88842a50747c900fedbfd8c3af1d90bdc"}
Nov 03 08:19:19.787983 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:19.787699    5070 scope.go:122] "RemoveContainer" containerID="b352c8ba7f3f94e3612f9c06f03b6efbd0508e20d4b388ee2bd4fddbfce28315"
Nov 03 08:19:20.071289 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.071256    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:19:20.104636 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.104605    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"]
Nov 03 08:19:20.105725 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.105702    5070 cpu_manager.go:414] "containerMap: removing container" podUID="dadc8621-5a99-4c99-a41d-a10cd5a73440" containerName="revoke-keys"
Nov 03 08:19:20.105902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.105727    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dadc8621-5a99-4c99-a41d-a10cd5a73440" containerName="revoke-keys"
Nov 03 08:19:20.105902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.105752    5070 cpu_manager.go:414] "containerMap: removing container" podUID="ae54e276-2450-4cef-98d9-e6cc85ed028a" containerName="istio-proxy"
Nov 03 08:19:20.105902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.105784    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae54e276-2450-4cef-98d9-e6cc85ed028a" containerName="istio-proxy"
Nov 03 08:19:20.105902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.105861    5070 cpu_manager.go:414] "containerMap: removing container" podUID="ae54e276-2450-4cef-98d9-e6cc85ed028a" containerName="istio-proxy"
Nov 03 08:19:20.105902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.105869    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae54e276-2450-4cef-98d9-e6cc85ed028a" containerName="istio-proxy"
Nov 03 08:19:20.106925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.106085    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="dadc8621-5a99-4c99-a41d-a10cd5a73440" containerName="revoke-keys"
Nov 03 08:19:20.106925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.106107    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae54e276-2450-4cef-98d9-e6cc85ed028a" containerName="istio-proxy"
Nov 03 08:19:20.106925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.106132    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae54e276-2450-4cef-98d9-e6cc85ed028a" containerName="istio-proxy"
Nov 03 08:19:20.110646 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.110621    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.126463 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.123702    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"]
Nov 03 08:19:20.145693 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.143845    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0fc79b-openshift-default-dockercfg-kmb5n\""
Nov 03 08:19:20.217796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.217752    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-credential-socket\") pod \"ae54e276-2450-4cef-98d9-e6cc85ed028a\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") "
Nov 03 08:19:20.217939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.217810    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-token\" (UniqueName: \"kubernetes.io/projected/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-token\") pod \"ae54e276-2450-4cef-98d9-e6cc85ed028a\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") "
Nov 03 08:19:20.217982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.217941    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-podinfo\") pod \"ae54e276-2450-4cef-98d9-e6cc85ed028a\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") "
Nov 03 08:19:20.218060 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.218043    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae54e276-2450-4cef-98d9-e6cc85ed028a-kube-api-access-wht9m\" (UniqueName: \"kubernetes.io/projected/ae54e276-2450-4cef-98d9-e6cc85ed028a-kube-api-access-wht9m\") pod \"ae54e276-2450-4cef-98d9-e6cc85ed028a\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") "
Nov 03 08:19:20.218186 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.218127    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae54e276-2450-4cef-98d9-e6cc85ed028a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae54e276-2450-4cef-98d9-e6cc85ed028a-istiod-ca-cert\") pod \"ae54e276-2450-4cef-98d9-e6cc85ed028a\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") "
Nov 03 08:19:20.218186 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.218129    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-credential-socket" pod "ae54e276-2450-4cef-98d9-e6cc85ed028a" (UID: "ae54e276-2450-4cef-98d9-e6cc85ed028a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:19:20.218186 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.218174    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-workload-socket\") pod \"ae54e276-2450-4cef-98d9-e6cc85ed028a\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") "
Nov 03 08:19:20.219232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.218201    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-data\") pod \"ae54e276-2450-4cef-98d9-e6cc85ed028a\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") "
Nov 03 08:19:20.219232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.218394    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-workload-certs\") pod \"ae54e276-2450-4cef-98d9-e6cc85ed028a\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") "
Nov 03 08:19:20.219232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.218402    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-workload-socket" pod "ae54e276-2450-4cef-98d9-e6cc85ed028a" (UID: "ae54e276-2450-4cef-98d9-e6cc85ed028a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:19:20.219232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.218491    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-envoy\") pod \"ae54e276-2450-4cef-98d9-e6cc85ed028a\" (UID: \"ae54e276-2450-4cef-98d9-e6cc85ed028a\") "
Nov 03 08:19:20.219232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.218586    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-data" pod "ae54e276-2450-4cef-98d9-e6cc85ed028a" (UID: "ae54e276-2450-4cef-98d9-e6cc85ed028a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:19:20.219232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.218596    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-workload-certs" pod "ae54e276-2450-4cef-98d9-e6cc85ed028a" (UID: "ae54e276-2450-4cef-98d9-e6cc85ed028a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:19:20.219232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.218624    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae54e276-2450-4cef-98d9-e6cc85ed028a-istiod-ca-cert" pod "ae54e276-2450-4cef-98d9-e6cc85ed028a" (UID: "ae54e276-2450-4cef-98d9-e6cc85ed028a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:19:20.219232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.218905    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdlgr\" (UniqueName: \"kubernetes.io/projected/79b2069b-a7c8-45eb-9739-9bf3a1fec044-kube-api-access-qdlgr\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.219232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.218941    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-podinfo\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.219232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.219085    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-workload-certs\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.219232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.219151    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-token\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.219944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.219246    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-envoy\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.219944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.219272    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-data\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.219944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.219402    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-workload-socket\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.219944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.219436    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-credential-socket\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.219944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.219486    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istiod-ca-cert\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.219944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.219693    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:20.219944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.219711    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae54e276-2450-4cef-98d9-e6cc85ed028a-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:20.219944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.219726    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:20.219944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.219742    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:20.219944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.219800    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:20.224877 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.224850    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae54e276-2450-4cef-98d9-e6cc85ed028a-kube-api-access-wht9m" pod "ae54e276-2450-4cef-98d9-e6cc85ed028a" (UID: "ae54e276-2450-4cef-98d9-e6cc85ed028a"). InnerVolumeSpecName "kube-api-access-wht9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:19:20.224973 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.224889    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-token" pod "ae54e276-2450-4cef-98d9-e6cc85ed028a" (UID: "ae54e276-2450-4cef-98d9-e6cc85ed028a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:19:20.225230 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.225210    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-envoy" pod "ae54e276-2450-4cef-98d9-e6cc85ed028a" (UID: "ae54e276-2450-4cef-98d9-e6cc85ed028a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:19:20.225284 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.225226    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-podinfo" pod "ae54e276-2450-4cef-98d9-e6cc85ed028a" (UID: "ae54e276-2450-4cef-98d9-e6cc85ed028a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:19:20.320865 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.320806    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-token\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.321082 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.320879    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-envoy\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.321082 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.320901    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-data\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.321082 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.320954    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-workload-socket\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.321082 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.321058    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-credential-socket\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.321354 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.321106    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istiod-ca-cert\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.321354 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.321304    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-workload-socket\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.322290 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.321360    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdlgr\" (UniqueName: \"kubernetes.io/projected/79b2069b-a7c8-45eb-9739-9bf3a1fec044-kube-api-access-qdlgr\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.322290 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.321386    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-podinfo\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.322290 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.321463    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-workload-certs\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.322290 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.321549    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:20.322290 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.321559    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wht9m\" (UniqueName: \"kubernetes.io/projected/ae54e276-2450-4cef-98d9-e6cc85ed028a-kube-api-access-wht9m\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:20.322290 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.321572    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:20.322290 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.321581    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae54e276-2450-4cef-98d9-e6cc85ed028a-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:20.322290 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.321749    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-data\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.322290 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.321892    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-workload-certs\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.322290 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.321890    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-credential-socket\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.324052 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.322989    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istiod-ca-cert\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.324750 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.324154    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-podinfo\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.324750 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.324256    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-envoy\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.327695 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.327673    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-token\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.328011 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.327993    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdlgr\" (UniqueName: \"kubernetes.io/projected/79b2069b-a7c8-45eb-9739-9bf3a1fec044-kube-api-access-qdlgr\") pod \"e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") " pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.446039 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.446012    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:20.798014 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.797980    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"
Nov 03 08:19:20.798462 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.798017    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k" event={"ID":"ae54e276-2450-4cef-98d9-e6cc85ed028a","Type":"ContainerDied","Data":"e049643897188cf5e03d0618d7dd9d68476592b375aea666b2d1ab505f19a17d"}
Nov 03 08:19:20.798462 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.798079    5070 scope.go:122] "RemoveContainer" containerID="75c299d56fbbae5553bf5fddbbc344e88842a50747c900fedbfd8c3af1d90bdc"
Nov 03 08:19:20.828360 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.828335    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"]
Nov 03 08:19:20.833084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.833061    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-99410b-openshift-default-545c6598d5-z2f7k"]
Nov 03 08:19:20.849361 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.849302    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"]
Nov 03 08:19:20.857619 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.857567    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:19:20.858620 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.857717    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:19:20.858620 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:20.857828    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:19:21.809524 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:21.809487    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg" event={"ID":"79b2069b-a7c8-45eb-9739-9bf3a1fec044","Type":"ContainerStarted","Data":"2494db5b7c62f02430d1dbc9a883e4ccd091db6f6c76ab1d0159d8b0d5dc8c3f"}
Nov 03 08:19:21.809977 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:21.809546    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg" event={"ID":"79b2069b-a7c8-45eb-9739-9bf3a1fec044","Type":"ContainerStarted","Data":"6e90a6ff9ccfa9fcd68bc714061595564feb7ed103a7e6b617b2928a1799457d"}
Nov 03 08:19:21.828240 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:21.828192    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg" podStartSLOduration=6.828173058 podStartE2EDuration="6.828173058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:19:21.826536567 +0000 UTC m=+1559.941000497" watchObservedRunningTime="2026-11-03 08:19:21.828173058 +0000 UTC m=+1559.942636990"
Nov 03 08:19:22.066373 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:22.066282    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae54e276-2450-4cef-98d9-e6cc85ed028a" path="/var/lib/kubelet/pods/ae54e276-2450-4cef-98d9-e6cc85ed028a/volumes"
Nov 03 08:19:22.086820 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:19:22.086748    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:19:22.446667 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:22.446637    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:22.453367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:22.453343    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:22.818184 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:22.818100    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:22.819030 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:22.819010    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:19:23.037085 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:23.037058    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"]
Nov 03 08:19:23.037259 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:23.037219    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz" podUID="337a39e3-b285-4d8c-aa1e-210a6e46763a" containerName="istio-proxy" containerID="cri-o://6bcc3636f20cc60c05dc615e934d95113699f8600830fd3ced73a7671d87d9b7" gracePeriod=30
Nov 03 08:19:28.408489 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.408456    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:19:28.442073 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.442042    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"]
Nov 03 08:19:28.442972 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.442952    5070 cpu_manager.go:414] "containerMap: removing container" podUID="337a39e3-b285-4d8c-aa1e-210a6e46763a" containerName="istio-proxy"
Nov 03 08:19:28.442972 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.442973    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="337a39e3-b285-4d8c-aa1e-210a6e46763a" containerName="istio-proxy"
Nov 03 08:19:28.443174 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.443150    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="337a39e3-b285-4d8c-aa1e-210a6e46763a" containerName="istio-proxy"
Nov 03 08:19:28.447081 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.447062    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.462090 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.462065    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"]
Nov 03 08:19:28.565196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.555425    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-workload-socket\") pod \"337a39e3-b285-4d8c-aa1e-210a6e46763a\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") "
Nov 03 08:19:28.565196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.555565    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-workload-certs\") pod \"337a39e3-b285-4d8c-aa1e-210a6e46763a\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") "
Nov 03 08:19:28.565196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.555601    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-envoy\") pod \"337a39e3-b285-4d8c-aa1e-210a6e46763a\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") "
Nov 03 08:19:28.565196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.555661    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-token\" (UniqueName: \"kubernetes.io/projected/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-token\") pod \"337a39e3-b285-4d8c-aa1e-210a6e46763a\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") "
Nov 03 08:19:28.565196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.555816    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-podinfo\") pod \"337a39e3-b285-4d8c-aa1e-210a6e46763a\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") "
Nov 03 08:19:28.565196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.555926    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/337a39e3-b285-4d8c-aa1e-210a6e46763a-kube-api-access-jmj52\" (UniqueName: \"kubernetes.io/projected/337a39e3-b285-4d8c-aa1e-210a6e46763a-kube-api-access-jmj52\") pod \"337a39e3-b285-4d8c-aa1e-210a6e46763a\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") "
Nov 03 08:19:28.565196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.556003    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/337a39e3-b285-4d8c-aa1e-210a6e46763a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337a39e3-b285-4d8c-aa1e-210a6e46763a-istiod-ca-cert\") pod \"337a39e3-b285-4d8c-aa1e-210a6e46763a\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") "
Nov 03 08:19:28.565196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.556185    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-data\") pod \"337a39e3-b285-4d8c-aa1e-210a6e46763a\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") "
Nov 03 08:19:28.565196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.556305    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-credential-socket\") pod \"337a39e3-b285-4d8c-aa1e-210a6e46763a\" (UID: \"337a39e3-b285-4d8c-aa1e-210a6e46763a\") "
Nov 03 08:19:28.565196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.556875    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2g6t\" (UniqueName: \"kubernetes.io/projected/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-kube-api-access-d2g6t\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.565196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.556928    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-workload-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.565196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.556974    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-credential-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.565196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.557214    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-data\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.565196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.557243    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-envoy\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.565196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.557443    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-workload-certs\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.565196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.557550    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-podinfo\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.565196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.557596    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-token\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.565196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.557650    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istiod-ca-cert\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.573267 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.573230    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-workload-socket" pod "337a39e3-b285-4d8c-aa1e-210a6e46763a" (UID: "337a39e3-b285-4d8c-aa1e-210a6e46763a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:19:28.573868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.573836    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-credential-socket" pod "337a39e3-b285-4d8c-aa1e-210a6e46763a" (UID: "337a39e3-b285-4d8c-aa1e-210a6e46763a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:19:28.579236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.579080    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-data" pod "337a39e3-b285-4d8c-aa1e-210a6e46763a" (UID: "337a39e3-b285-4d8c-aa1e-210a6e46763a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:19:28.579236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.579212    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/337a39e3-b285-4d8c-aa1e-210a6e46763a-istiod-ca-cert" pod "337a39e3-b285-4d8c-aa1e-210a6e46763a" (UID: "337a39e3-b285-4d8c-aa1e-210a6e46763a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:19:28.579643 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.579624    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-workload-certs" pod "337a39e3-b285-4d8c-aa1e-210a6e46763a" (UID: "337a39e3-b285-4d8c-aa1e-210a6e46763a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:19:28.579839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.579812    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-envoy" pod "337a39e3-b285-4d8c-aa1e-210a6e46763a" (UID: "337a39e3-b285-4d8c-aa1e-210a6e46763a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:19:28.580135 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.580091    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-podinfo" pod "337a39e3-b285-4d8c-aa1e-210a6e46763a" (UID: "337a39e3-b285-4d8c-aa1e-210a6e46763a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:19:28.609264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.609238    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337a39e3-b285-4d8c-aa1e-210a6e46763a-kube-api-access-jmj52" pod "337a39e3-b285-4d8c-aa1e-210a6e46763a" (UID: "337a39e3-b285-4d8c-aa1e-210a6e46763a"). InnerVolumeSpecName "kube-api-access-jmj52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:19:28.609618 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.609591    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-token" pod "337a39e3-b285-4d8c-aa1e-210a6e46763a" (UID: "337a39e3-b285-4d8c-aa1e-210a6e46763a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:19:28.660038 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.659956    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-data\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.660038 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.660001    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-envoy\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.660239 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.660117    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-workload-certs\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.660239 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.660189    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-podinfo\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.660239 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.660217    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-token\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.660347 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.660254    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istiod-ca-cert\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.660347 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.660337    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2g6t\" (UniqueName: \"kubernetes.io/projected/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-kube-api-access-d2g6t\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.660430 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.660370    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-workload-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.660430 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.660399    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-credential-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.661930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.660627    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:28.661930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.660640    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:28.661930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.660650    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:28.661930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.660665    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:28.661930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.660679    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:28.661930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.660689    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:28.661930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.660700    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/337a39e3-b285-4d8c-aa1e-210a6e46763a-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:28.661930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.660712    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmj52\" (UniqueName: \"kubernetes.io/projected/337a39e3-b285-4d8c-aa1e-210a6e46763a-kube-api-access-jmj52\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:28.661930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.660721    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/337a39e3-b285-4d8c-aa1e-210a6e46763a-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:19:28.661930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.661133    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-credential-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.661930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.661378    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-data\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.663406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.661932    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istiod-ca-cert\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.663406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.662153    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-workload-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.663406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.662198    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-workload-certs\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.664999 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.664975    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-envoy\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.666033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.666006    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-podinfo\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.670502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.670476    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2g6t\" (UniqueName: \"kubernetes.io/projected/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-kube-api-access-d2g6t\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.674223 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.674200    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-token\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.766278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.766253    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:28.968331 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.959287    5070 generic.go:356] "Generic (PLEG): container finished" podID="337a39e3-b285-4d8c-aa1e-210a6e46763a" containerID="6bcc3636f20cc60c05dc615e934d95113699f8600830fd3ced73a7671d87d9b7" exitCode=0
Nov 03 08:19:28.968331 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.959540    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz" event={"ID":"337a39e3-b285-4d8c-aa1e-210a6e46763a","Type":"ContainerDied","Data":"6bcc3636f20cc60c05dc615e934d95113699f8600830fd3ced73a7671d87d9b7"}
Nov 03 08:19:28.968331 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.959636    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz" event={"ID":"337a39e3-b285-4d8c-aa1e-210a6e46763a","Type":"ContainerDied","Data":"b330ad09eed9c381494780ec2692b2b4579f05c627a9401d69240cff675095a8"}
Nov 03 08:19:28.968331 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.959669    5070 scope.go:122] "RemoveContainer" containerID="6bcc3636f20cc60c05dc615e934d95113699f8600830fd3ced73a7671d87d9b7"
Nov 03 08:19:28.968331 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.959865    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"
Nov 03 08:19:28.989748 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.989715    5070 scope.go:122] "RemoveContainer" containerID="6bcc3636f20cc60c05dc615e934d95113699f8600830fd3ced73a7671d87d9b7"
Nov 03 08:19:28.994974 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:19:28.994890    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bcc3636f20cc60c05dc615e934d95113699f8600830fd3ced73a7671d87d9b7\": container with ID starting with 6bcc3636f20cc60c05dc615e934d95113699f8600830fd3ced73a7671d87d9b7 not found: ID does not exist" containerID="6bcc3636f20cc60c05dc615e934d95113699f8600830fd3ced73a7671d87d9b7"
Nov 03 08:19:28.994974 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.994932    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bcc3636f20cc60c05dc615e934d95113699f8600830fd3ced73a7671d87d9b7"} err="failed to get container status \"6bcc3636f20cc60c05dc615e934d95113699f8600830fd3ced73a7671d87d9b7\": rpc error: code = NotFound desc = could not find container \"6bcc3636f20cc60c05dc615e934d95113699f8600830fd3ced73a7671d87d9b7\": container with ID starting with 6bcc3636f20cc60c05dc615e934d95113699f8600830fd3ced73a7671d87d9b7 not found: ID does not exist"
Nov 03 08:19:28.999131 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:28.999096    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"]
Nov 03 08:19:29.006861 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:29.006831    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-tlbnz"]
Nov 03 08:19:29.057356 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:29.057328    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"]
Nov 03 08:19:29.058811 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:19:29.058784    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c9e812_cff9_46b7_8ee1_1bea23dcd7fd.slice/crio-4c79800a66cf207083df2a35af46cc33f9ff50b7d6d2d13272023079fd5999a9 WatchSource:0}: Error finding container 4c79800a66cf207083df2a35af46cc33f9ff50b7d6d2d13272023079fd5999a9: Status 404 returned error can't find the container with id 4c79800a66cf207083df2a35af46cc33f9ff50b7d6d2d13272023079fd5999a9
Nov 03 08:19:29.061658 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:29.061625    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:19:29.061725 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:29.061690    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:19:29.061725 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:29.061721    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:19:29.969559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:29.969522    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx" event={"ID":"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd","Type":"ContainerStarted","Data":"a9cf4d155950d607fdf623e6e220b88a4bc525f0c17872ffcc38eea4eca176c3"}
Nov 03 08:19:29.970672 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:29.969582    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx" event={"ID":"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd","Type":"ContainerStarted","Data":"4c79800a66cf207083df2a35af46cc33f9ff50b7d6d2d13272023079fd5999a9"}
Nov 03 08:19:29.990972 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:29.990926    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx" podStartSLOduration=40.990912099 podStartE2EDuration="40.990912099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:18:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:19:29.986748181 +0000 UTC m=+1568.101212112" watchObservedRunningTime="2026-11-03 08:19:29.990912099 +0000 UTC m=+1568.105376030"
Nov 03 08:19:30.028185 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:30.028158    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="337a39e3-b285-4d8c-aa1e-210a6e46763a" path="/var/lib/kubelet/pods/337a39e3-b285-4d8c-aa1e-210a6e46763a/volumes"
Nov 03 08:19:30.780780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:30.776168    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:30.781108 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:30.781077    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 03 08:19:30.781324 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:30.781298    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx" podUID="a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 03 08:19:31.775781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:31.775050    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 03 08:19:31.775781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:31.775106    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx" podUID="a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 03 08:19:32.771164 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:32.771120    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 03 08:19:32.771467 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:32.771435    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx" podUID="a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 03 08:19:33.778074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:33.778030    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 03 08:19:33.778585 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:33.778095    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx" podUID="a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 03 08:19:34.780796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:34.775053    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 03 08:19:34.780796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:34.775110    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx" podUID="a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 03 08:19:35.821783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:35.817862    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Nov 03 08:19:35.821783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:35.817923    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx" podUID="a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Nov 03 08:19:36.772044 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:36.772016    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:36.772370 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:36.772349    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:36.772854 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:36.772835    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:19:42.213667 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:19:42.213635    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:19:52.052406 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:19:52.052370    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:20:13.875800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:13.875754    5070 scope.go:122] "RemoveContainer" containerID="04e1e47da8e5b4144e964a513167699f2b9279b527163f28bde1351a49c1cffb"
Nov 03 08:20:13.888393 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:13.888373    5070 scope.go:122] "RemoveContainer" containerID="23005b5dbc4e80e62b6ae5e46757a40b93769b4e96bf189924ca7104dc752464"
Nov 03 08:20:13.900525 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:13.900505    5070 scope.go:122] "RemoveContainer" containerID="349234bc369fa4aea009b05985d7d0d87fe929a99f81a9eafe4acf522c95a9b6"
Nov 03 08:20:22.052356 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:20:22.052324    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:20:32.210938 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:32.210906    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95"]
Nov 03 08:20:32.217610 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:32.217584    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95"
Nov 03 08:20:32.221033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:32.221010    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:20:32.221182 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:32.221146    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:20:32.221290 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:32.221259    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:20:32.221353 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:32.221299    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:20:32.223437 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:32.223412    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95"]
Nov 03 08:20:32.337567 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:32.337537    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb13cee-7c55-4be0-b9de-a65d10cd693f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-27c674-rbb95\" (UID: \"7bb13cee-7c55-4be0-b9de-a65d10cd693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95"
Nov 03 08:20:32.337726 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:32.337578    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb13cee-7c55-4be0-b9de-a65d10cd693f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-27c674-rbb95\" (UID: \"7bb13cee-7c55-4be0-b9de-a65d10cd693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95"
Nov 03 08:20:32.337726 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:32.337605    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrvzk\" (UniqueName: \"kubernetes.io/projected/7bb13cee-7c55-4be0-b9de-a65d10cd693f-kube-api-access-zrvzk\") pod \"maas-api-revoke-keys-e2e-inf-a-27c674-rbb95\" (UID: \"7bb13cee-7c55-4be0-b9de-a65d10cd693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95"
Nov 03 08:20:32.439216 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:32.439182    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb13cee-7c55-4be0-b9de-a65d10cd693f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-27c674-rbb95\" (UID: \"7bb13cee-7c55-4be0-b9de-a65d10cd693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95"
Nov 03 08:20:32.439369 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:32.439227    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb13cee-7c55-4be0-b9de-a65d10cd693f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-27c674-rbb95\" (UID: \"7bb13cee-7c55-4be0-b9de-a65d10cd693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95"
Nov 03 08:20:32.439369 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:32.439250    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrvzk\" (UniqueName: \"kubernetes.io/projected/7bb13cee-7c55-4be0-b9de-a65d10cd693f-kube-api-access-zrvzk\") pod \"maas-api-revoke-keys-e2e-inf-a-27c674-rbb95\" (UID: \"7bb13cee-7c55-4be0-b9de-a65d10cd693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95"
Nov 03 08:20:32.440482 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:32.439962    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb13cee-7c55-4be0-b9de-a65d10cd693f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-27c674-rbb95\" (UID: \"7bb13cee-7c55-4be0-b9de-a65d10cd693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95"
Nov 03 08:20:32.441581 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:32.441558    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb13cee-7c55-4be0-b9de-a65d10cd693f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-27c674-rbb95\" (UID: \"7bb13cee-7c55-4be0-b9de-a65d10cd693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95"
Nov 03 08:20:32.461871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:32.461810    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrvzk\" (UniqueName: \"kubernetes.io/projected/7bb13cee-7c55-4be0-b9de-a65d10cd693f-kube-api-access-zrvzk\") pod \"maas-api-revoke-keys-e2e-inf-a-27c674-rbb95\" (UID: \"7bb13cee-7c55-4be0-b9de-a65d10cd693f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95"
Nov 03 08:20:32.543161 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:32.543070    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95"
Nov 03 08:20:33.775303 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:33.775258    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded"
Nov 03 08:20:33.862203 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:33.801353    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Nov 03 08:20:33.888126 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:33.801404    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:20:34.162829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:34.146658    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:20:34.271234 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:34.269862    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:20:34.322707 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:34.322660    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:20:34.492687 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:34.388855    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:20:34.522798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:34.513881    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:20:34.814221 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:34.814186    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95"]
Nov 03 08:20:35.677734 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:35.677699    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95" event={"ID":"7bb13cee-7c55-4be0-b9de-a65d10cd693f","Type":"ContainerStarted","Data":"053a383cb4665a3cbc082e68cfaa70d901567e6b41f8e2a0dbfe49950e1d6d28"}
Nov 03 08:20:35.678028 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:35.678009    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95" event={"ID":"7bb13cee-7c55-4be0-b9de-a65d10cd693f","Type":"ContainerStarted","Data":"e2716300924f39585c349888c8aca9d67b1ce53125a45cf6e1e26778c71144f5"}
Nov 03 08:20:35.715997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:35.714689    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95" podStartSLOduration=3.714670971 podStartE2EDuration="3.714670971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:20:35.710203804 +0000 UTC m=+1633.824667736" watchObservedRunningTime="2026-11-03 08:20:35.714670971 +0000 UTC m=+1633.829134904"
Nov 03 08:20:36.705512 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:36.705471    5070 generic.go:356] "Generic (PLEG): container finished" podID="7bb13cee-7c55-4be0-b9de-a65d10cd693f" containerID="053a383cb4665a3cbc082e68cfaa70d901567e6b41f8e2a0dbfe49950e1d6d28" exitCode=0
Nov 03 08:20:36.706061 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:36.705553    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95" event={"ID":"7bb13cee-7c55-4be0-b9de-a65d10cd693f","Type":"ContainerDied","Data":"053a383cb4665a3cbc082e68cfaa70d901567e6b41f8e2a0dbfe49950e1d6d28"}
Nov 03 08:20:37.949180 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:37.949155    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95"
Nov 03 08:20:38.018065 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:38.017686    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bb13cee-7c55-4be0-b9de-a65d10cd693f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb13cee-7c55-4be0-b9de-a65d10cd693f-maas-api-xks-ca\") pod \"7bb13cee-7c55-4be0-b9de-a65d10cd693f\" (UID: \"7bb13cee-7c55-4be0-b9de-a65d10cd693f\") "
Nov 03 08:20:38.018065 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:38.017951    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bb13cee-7c55-4be0-b9de-a65d10cd693f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb13cee-7c55-4be0-b9de-a65d10cd693f-maas-api-openshift-ca\") pod \"7bb13cee-7c55-4be0-b9de-a65d10cd693f\" (UID: \"7bb13cee-7c55-4be0-b9de-a65d10cd693f\") "
Nov 03 08:20:38.018314 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:38.018160    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bb13cee-7c55-4be0-b9de-a65d10cd693f-kube-api-access-zrvzk\" (UniqueName: \"kubernetes.io/projected/7bb13cee-7c55-4be0-b9de-a65d10cd693f-kube-api-access-zrvzk\") pod \"7bb13cee-7c55-4be0-b9de-a65d10cd693f\" (UID: \"7bb13cee-7c55-4be0-b9de-a65d10cd693f\") "
Nov 03 08:20:38.018384 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:38.018344    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb13cee-7c55-4be0-b9de-a65d10cd693f-maas-api-openshift-ca" pod "7bb13cee-7c55-4be0-b9de-a65d10cd693f" (UID: "7bb13cee-7c55-4be0-b9de-a65d10cd693f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:20:38.019026 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:38.018997    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb13cee-7c55-4be0-b9de-a65d10cd693f-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:20:38.020449 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:38.020420    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb13cee-7c55-4be0-b9de-a65d10cd693f-maas-api-xks-ca" pod "7bb13cee-7c55-4be0-b9de-a65d10cd693f" (UID: "7bb13cee-7c55-4be0-b9de-a65d10cd693f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:20:38.020715 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:38.020686    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb13cee-7c55-4be0-b9de-a65d10cd693f-kube-api-access-zrvzk" pod "7bb13cee-7c55-4be0-b9de-a65d10cd693f" (UID: "7bb13cee-7c55-4be0-b9de-a65d10cd693f"). InnerVolumeSpecName "kube-api-access-zrvzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:20:38.207347 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:38.206940    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrvzk\" (UniqueName: \"kubernetes.io/projected/7bb13cee-7c55-4be0-b9de-a65d10cd693f-kube-api-access-zrvzk\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:20:38.207347 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:38.206981    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb13cee-7c55-4be0-b9de-a65d10cd693f-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:20:38.726873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:38.726825    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95" event={"ID":"7bb13cee-7c55-4be0-b9de-a65d10cd693f","Type":"ContainerDied","Data":"e2716300924f39585c349888c8aca9d67b1ce53125a45cf6e1e26778c71144f5"}
Nov 03 08:20:38.727047 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:38.726892    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2716300924f39585c349888c8aca9d67b1ce53125a45cf6e1e26778c71144f5"
Nov 03 08:20:38.727047 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:38.726955    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95"
Nov 03 08:20:44.778781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:44.773189    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95"]
Nov 03 08:20:44.794796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:44.784547    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-27c674-rbb95"]
Nov 03 08:20:46.029189 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:46.029156    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bb13cee-7c55-4be0-b9de-a65d10cd693f" path="/var/lib/kubelet/pods/7bb13cee-7c55-4be0-b9de-a65d10cd693f/volumes"
Nov 03 08:20:52.219386 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:20:52.219326    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:20:52.327918 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:52.327876    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"]
Nov 03 08:20:52.951053 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:52.328828    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws" podUID="5d3803b1-7a4f-4e73-b541-9cb0c27c624d" containerName="istio-proxy" containerID="cri-o://b7d9e635028bca7e1d91ffdc288d9cbb50c6df1cad4b6277f7bc44e53d897936" gracePeriod=30
Nov 03 08:20:52.951053 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:52.910821    5070 patch_prober.go:29] interesting pod/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": read tcp 10.132.0.2:45722->10.132.0.215:15021: read: connection reset by peer" start-of-body=
Nov 03 08:20:52.951053 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:52.910888    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg" podUID="79b2069b-a7c8-45eb-9739-9bf3a1fec044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": read tcp 10.132.0.2:45722->10.132.0.215:15021: read: connection reset by peer"
Nov 03 08:20:53.180129 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:53.180048    5070 generic.go:356] "Generic (PLEG): container finished" podID="79b2069b-a7c8-45eb-9739-9bf3a1fec044" containerID="2494db5b7c62f02430d1dbc9a883e4ccd091db6f6c76ab1d0159d8b0d5dc8c3f" exitCode=0
Nov 03 08:20:53.180370 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:53.180122    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg" event={"ID":"79b2069b-a7c8-45eb-9739-9bf3a1fec044","Type":"ContainerDied","Data":"2494db5b7c62f02430d1dbc9a883e4ccd091db6f6c76ab1d0159d8b0d5dc8c3f"}
Nov 03 08:20:53.196806 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:53.180597    5070 scope.go:122] "RemoveContainer" containerID="2494db5b7c62f02430d1dbc9a883e4ccd091db6f6c76ab1d0159d8b0d5dc8c3f"
Nov 03 08:20:53.196806 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:53.183195    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:20:53.196806 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:53.183267    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:20:53.196806 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:53.183299    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:20:54.223304 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:54.223267    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg" event={"ID":"79b2069b-a7c8-45eb-9739-9bf3a1fec044","Type":"ContainerStarted","Data":"772ca4ffdb2c297a843bbf7b384c2a89d75821e380e310aeee6b4907ffe91c21"}
Nov 03 08:20:54.450340 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:54.448999    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:20:54.450340 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:54.449149    5070 patch_prober.go:29] interesting pod/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 03 08:20:54.450340 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:54.449204    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg" podUID="79b2069b-a7c8-45eb-9739-9bf3a1fec044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 03 08:20:55.447072 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:55.447038    5070 patch_prober.go:29] interesting pod/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 03 08:20:55.447572 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:55.447108    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg" podUID="79b2069b-a7c8-45eb-9739-9bf3a1fec044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 03 08:20:56.446981 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:56.446946    5070 patch_prober.go:29] interesting pod/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 03 08:20:56.447174 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:56.447002    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg" podUID="79b2069b-a7c8-45eb-9739-9bf3a1fec044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 03 08:20:57.454121 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:57.454085    5070 patch_prober.go:29] interesting pod/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 03 08:20:57.464462 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:57.463894    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg" podUID="79b2069b-a7c8-45eb-9739-9bf3a1fec044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 03 08:20:58.120900 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.119778    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:20:58.196832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.196790    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-credential-socket\") pod \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") "
Nov 03 08:20:58.197055 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.196897    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-envoy\") pod \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") "
Nov 03 08:20:58.197055 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.196993    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-podinfo\") pod \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") "
Nov 03 08:20:58.197189 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.197062    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-kube-api-access-ddb2t\" (UniqueName: \"kubernetes.io/projected/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-kube-api-access-ddb2t\") pod \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") "
Nov 03 08:20:58.197189 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.197136    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-data\") pod \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") "
Nov 03 08:20:58.197347 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.197325    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-workload-socket\") pod \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") "
Nov 03 08:20:58.198068 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.197379    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-token\" (UniqueName: \"kubernetes.io/projected/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-token\") pod \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") "
Nov 03 08:20:58.198068 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.197509    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istiod-ca-cert\") pod \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") "
Nov 03 08:20:58.198068 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.197556    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-workload-certs\") pod \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\" (UID: \"5d3803b1-7a4f-4e73-b541-9cb0c27c624d\") "
Nov 03 08:20:58.211565 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.211531    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-workload-socket" pod "5d3803b1-7a4f-4e73-b541-9cb0c27c624d" (UID: "5d3803b1-7a4f-4e73-b541-9cb0c27c624d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:20:58.211854 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.211779    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-credential-socket" pod "5d3803b1-7a4f-4e73-b541-9cb0c27c624d" (UID: "5d3803b1-7a4f-4e73-b541-9cb0c27c624d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:20:58.216241 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.216150    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-workload-certs" pod "5d3803b1-7a4f-4e73-b541-9cb0c27c624d" (UID: "5d3803b1-7a4f-4e73-b541-9cb0c27c624d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:20:58.218886 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.218853    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"]
Nov 03 08:20:58.220919 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.220893    5070 cpu_manager.go:414] "containerMap: removing container" podUID="5d3803b1-7a4f-4e73-b541-9cb0c27c624d" containerName="istio-proxy"
Nov 03 08:20:58.221015 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.220920    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3803b1-7a4f-4e73-b541-9cb0c27c624d" containerName="istio-proxy"
Nov 03 08:20:58.221450 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.221427    5070 cpu_manager.go:414] "containerMap: removing container" podUID="7bb13cee-7c55-4be0-b9de-a65d10cd693f" containerName="revoke-keys"
Nov 03 08:20:58.221542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.221452    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bb13cee-7c55-4be0-b9de-a65d10cd693f" containerName="revoke-keys"
Nov 03 08:20:58.224929 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.224904    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bb13cee-7c55-4be0-b9de-a65d10cd693f" containerName="revoke-keys"
Nov 03 08:20:58.225030 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.224956    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d3803b1-7a4f-4e73-b541-9cb0c27c624d" containerName="istio-proxy"
Nov 03 08:20:58.236917 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.230026    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-kube-api-access-ddb2t" pod "5d3803b1-7a4f-4e73-b541-9cb0c27c624d" (UID: "5d3803b1-7a4f-4e73-b541-9cb0c27c624d"). InnerVolumeSpecName "kube-api-access-ddb2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:20:58.236917 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.231519    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istiod-ca-cert" pod "5d3803b1-7a4f-4e73-b541-9cb0c27c624d" (UID: "5d3803b1-7a4f-4e73-b541-9cb0c27c624d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:20:58.236917 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.232148    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-data" pod "5d3803b1-7a4f-4e73-b541-9cb0c27c624d" (UID: "5d3803b1-7a4f-4e73-b541-9cb0c27c624d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:20:58.249839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.244664    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-envoy" pod "5d3803b1-7a4f-4e73-b541-9cb0c27c624d" (UID: "5d3803b1-7a4f-4e73-b541-9cb0c27c624d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:20:58.249839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.247890    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-podinfo" pod "5d3803b1-7a4f-4e73-b541-9cb0c27c624d" (UID: "5d3803b1-7a4f-4e73-b541-9cb0c27c624d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:20:58.249839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.247933    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-token" pod "5d3803b1-7a4f-4e73-b541-9cb0c27c624d" (UID: "5d3803b1-7a4f-4e73-b541-9cb0c27c624d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:20:58.312712 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.310711    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"]
Nov 03 08:20:58.312712 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.310981    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.315799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.314603    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-109070-openshift-default-dockercfg-5wqds\""
Nov 03 08:20:58.317682 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.317652    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:20:58.317872 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.317853    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddb2t\" (UniqueName: \"kubernetes.io/projected/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-kube-api-access-ddb2t\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:20:58.317991 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.317977    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:20:58.318093 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.318081    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:20:58.318179 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.318168    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:20:58.318277 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.318264    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:20:58.318369 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.318357    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:20:58.318458 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.318446    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:20:58.318547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.318536    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d3803b1-7a4f-4e73-b541-9cb0c27c624d-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:20:58.326450 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.326423    5070 generic.go:356] "Generic (PLEG): container finished" podID="5d3803b1-7a4f-4e73-b541-9cb0c27c624d" containerID="b7d9e635028bca7e1d91ffdc288d9cbb50c6df1cad4b6277f7bc44e53d897936" exitCode=0
Nov 03 08:20:58.326723 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.326705    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"
Nov 03 08:20:58.327114 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.327091    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws" event={"ID":"5d3803b1-7a4f-4e73-b541-9cb0c27c624d","Type":"ContainerDied","Data":"b7d9e635028bca7e1d91ffdc288d9cbb50c6df1cad4b6277f7bc44e53d897936"}
Nov 03 08:20:58.327227 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.327215    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws" event={"ID":"5d3803b1-7a4f-4e73-b541-9cb0c27c624d","Type":"ContainerDied","Data":"778e63d0bb7052b080e429cd50344bddc679612b183e6f3cb2d517ad4fca7e54"}
Nov 03 08:20:58.327302 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.327293    5070 scope.go:122] "RemoveContainer" containerID="b7d9e635028bca7e1d91ffdc288d9cbb50c6df1cad4b6277f7bc44e53d897936"
Nov 03 08:20:58.349691 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.349669    5070 scope.go:122] "RemoveContainer" containerID="b7d9e635028bca7e1d91ffdc288d9cbb50c6df1cad4b6277f7bc44e53d897936"
Nov 03 08:20:58.350001 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:20:58.349974    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d9e635028bca7e1d91ffdc288d9cbb50c6df1cad4b6277f7bc44e53d897936\": container with ID starting with b7d9e635028bca7e1d91ffdc288d9cbb50c6df1cad4b6277f7bc44e53d897936 not found: ID does not exist" containerID="b7d9e635028bca7e1d91ffdc288d9cbb50c6df1cad4b6277f7bc44e53d897936"
Nov 03 08:20:58.350069 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.350014    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d9e635028bca7e1d91ffdc288d9cbb50c6df1cad4b6277f7bc44e53d897936"} err="failed to get container status \"b7d9e635028bca7e1d91ffdc288d9cbb50c6df1cad4b6277f7bc44e53d897936\": rpc error: code = NotFound desc = could not find container \"b7d9e635028bca7e1d91ffdc288d9cbb50c6df1cad4b6277f7bc44e53d897936\": container with ID starting with b7d9e635028bca7e1d91ffdc288d9cbb50c6df1cad4b6277f7bc44e53d897936 not found: ID does not exist"
Nov 03 08:20:58.387914 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.387880    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"]
Nov 03 08:20:58.392582 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.392558    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-27c674-openshift-default-7b8f474c4f-qd6ws"]
Nov 03 08:20:58.420548 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.420518    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-podinfo\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.420741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.420715    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-workload-socket\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.420832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.420815    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-envoy\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.420882 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.420872    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stb4t\" (UniqueName: \"kubernetes.io/projected/8e96f256-aa30-4c3e-9d97-d829c96d13ac-kube-api-access-stb4t\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.421014 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.420993    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-data\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.421139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.421113    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istiod-ca-cert\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.421195 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.421181    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-workload-certs\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.421248 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.421206    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-token\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.421306 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.421253    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-credential-socket\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.447271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.447250    5070 patch_prober.go:29] interesting pod/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 03 08:20:58.447377 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.447317    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg" podUID="79b2069b-a7c8-45eb-9739-9bf3a1fec044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 03 08:20:58.522735 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.522705    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istiod-ca-cert\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.523104 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.522801    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-workload-certs\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.523104 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.522819    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-token\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.523104 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.522841    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-credential-socket\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.523104 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.522916    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-podinfo\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.523104 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.523071    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-workload-socket\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.523389 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.523115    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-envoy\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.523389 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.523173    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-workload-certs\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.523389 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.523250    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stb4t\" (UniqueName: \"kubernetes.io/projected/8e96f256-aa30-4c3e-9d97-d829c96d13ac-kube-api-access-stb4t\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.523389 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.523278    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-credential-socket\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.523389 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.523335    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-data\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.523638 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.523431    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-workload-socket\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.523638 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.523488    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istiod-ca-cert\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.524998 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.523666    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-data\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.525370 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.525351    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-envoy\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.525976 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.525641    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-podinfo\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.530365 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.530340    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-token\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.531193 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.531171    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stb4t\" (UniqueName: \"kubernetes.io/projected/8e96f256-aa30-4c3e-9d97-d829c96d13ac-kube-api-access-stb4t\") pod \"e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") " pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:58.639438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:58.639352    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:20:59.112367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:59.112335    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"]
Nov 03 08:20:59.113026 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:20:59.112994    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e96f256_aa30_4c3e_9d97_d829c96d13ac.slice/crio-8f13227603bae4ee40deb029657293da112550ec4b0211d1830220a4ef39acf8 WatchSource:0}: Error finding container 8f13227603bae4ee40deb029657293da112550ec4b0211d1830220a4ef39acf8: Status 404 returned error can't find the container with id 8f13227603bae4ee40deb029657293da112550ec4b0211d1830220a4ef39acf8
Nov 03 08:20:59.116845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:59.116806    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:20:59.118146 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:59.116896    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:20:59.118146 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:59.116943    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:20:59.342678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:59.340876    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b" event={"ID":"8e96f256-aa30-4c3e-9d97-d829c96d13ac","Type":"ContainerStarted","Data":"f05d3136eb9be54b7c1b46679e84dab6c1c0ede5c0327ab9abf0ed279871202e"}
Nov 03 08:20:59.342678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:59.340956    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b" event={"ID":"8e96f256-aa30-4c3e-9d97-d829c96d13ac","Type":"ContainerStarted","Data":"8f13227603bae4ee40deb029657293da112550ec4b0211d1830220a4ef39acf8"}
Nov 03 08:20:59.386782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:59.383390    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b" podStartSLOduration=70.38336116 podStartE2EDuration="1m10.38336116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:20:59.378076674 +0000 UTC m=+1657.492540606" watchObservedRunningTime="2026-11-03 08:20:59.38336116 +0000 UTC m=+1657.497825091"
Nov 03 08:20:59.446724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:59.446638    5070 patch_prober.go:29] interesting pod/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 03 08:20:59.446724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:59.446692    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg" podUID="79b2069b-a7c8-45eb-9739-9bf3a1fec044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 03 08:20:59.641794 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:20:59.641195    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:21:00.027571 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:00.027536    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d3803b1-7a4f-4e73-b541-9cb0c27c624d" path="/var/lib/kubelet/pods/5d3803b1-7a4f-4e73-b541-9cb0c27c624d/volumes"
Nov 03 08:21:00.453781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:00.448278    5070 patch_prober.go:29] interesting pod/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Nov 03 08:21:00.453781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:00.448367    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg" podUID="79b2069b-a7c8-45eb-9739-9bf3a1fec044" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Nov 03 08:21:00.646949 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:00.646880    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:21:01.374809 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:01.366056    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:21:01.406789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:01.385291    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:21:01.499196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:01.492269    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:21:01.499196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:01.494150    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:21:01.506033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:01.504093    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:21:04.172621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:04.172541    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:21:13.957887 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:13.957856    5070 scope.go:122] "RemoveContainer" containerID="814bea5c0fbef0170dce997f3c620cc6c2902cf5aca0dd02f29ee7e4d25ab9c4"
Nov 03 08:21:17.433701 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:17.433666    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"]
Nov 03 08:21:17.434160 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:17.433879    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p" podUID="4f3b3503-699a-4697-b121-351ad81bad8a" containerName="istio-proxy" containerID="cri-o://41f9ed86772f153450c891f4cf3bbbd6ca5a5c633806e8d922f1621e3a16c7ef" gracePeriod=30
Nov 03 08:21:22.052936 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:21:22.052897    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:21:22.791350 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.791321    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:21:22.845447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.845413    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"]
Nov 03 08:21:22.846449 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.846423    5070 cpu_manager.go:414] "containerMap: removing container" podUID="4f3b3503-699a-4697-b121-351ad81bad8a" containerName="istio-proxy"
Nov 03 08:21:22.846648 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.846624    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f3b3503-699a-4697-b121-351ad81bad8a" containerName="istio-proxy"
Nov 03 08:21:22.846778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.846685    5070 cpu_manager.go:414] "containerMap: removing container" podUID="4f3b3503-699a-4697-b121-351ad81bad8a" containerName="istio-proxy"
Nov 03 08:21:22.846778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.846695    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f3b3503-699a-4697-b121-351ad81bad8a" containerName="istio-proxy"
Nov 03 08:21:22.846968 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.846954    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f3b3503-699a-4697-b121-351ad81bad8a" containerName="istio-proxy"
Nov 03 08:21:22.847013 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.846972    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f3b3503-699a-4697-b121-351ad81bad8a" containerName="istio-proxy"
Nov 03 08:21:22.851166 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.851145    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:22.853902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.853877    5070 generic.go:356] "Generic (PLEG): container finished" podID="4f3b3503-699a-4697-b121-351ad81bad8a" containerID="41f9ed86772f153450c891f4cf3bbbd6ca5a5c633806e8d922f1621e3a16c7ef" exitCode=0
Nov 03 08:21:22.854118 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.853908    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p" event={"ID":"4f3b3503-699a-4697-b121-351ad81bad8a","Type":"ContainerDied","Data":"41f9ed86772f153450c891f4cf3bbbd6ca5a5c633806e8d922f1621e3a16c7ef"}
Nov 03 08:21:22.854204 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.854150    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p" event={"ID":"4f3b3503-699a-4697-b121-351ad81bad8a","Type":"ContainerDied","Data":"89885ad42ff98109be87400735033af848932d78a9e326115506a6914e817e2a"}
Nov 03 08:21:22.854267 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.854187    5070 scope.go:122] "RemoveContainer" containerID="41f9ed86772f153450c891f4cf3bbbd6ca5a5c633806e8d922f1621e3a16c7ef"
Nov 03 08:21:22.854326 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.853947    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"
Nov 03 08:21:22.873693 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.873668    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"]
Nov 03 08:21:22.893567 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.893539    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f3b3503-699a-4697-b121-351ad81bad8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3b3503-699a-4697-b121-351ad81bad8a-istiod-ca-cert\") pod \"4f3b3503-699a-4697-b121-351ad81bad8a\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") "
Nov 03 08:21:22.893709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.893603    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-workload-socket\") pod \"4f3b3503-699a-4697-b121-351ad81bad8a\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") "
Nov 03 08:21:22.893709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.893684    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f3b3503-699a-4697-b121-351ad81bad8a-istio-token\" (UniqueName: \"kubernetes.io/projected/4f3b3503-699a-4697-b121-351ad81bad8a-istio-token\") pod \"4f3b3503-699a-4697-b121-351ad81bad8a\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") "
Nov 03 08:21:22.893968 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.893770    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-workload-certs\") pod \"4f3b3503-699a-4697-b121-351ad81bad8a\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") "
Nov 03 08:21:22.893968 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.893819    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-credential-socket\") pod \"4f3b3503-699a-4697-b121-351ad81bad8a\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") "
Nov 03 08:21:22.893968 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.893905    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-istio-envoy\") pod \"4f3b3503-699a-4697-b121-351ad81bad8a\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") "
Nov 03 08:21:22.894163 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.894060    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f3b3503-699a-4697-b121-351ad81bad8a-kube-api-access-qwfbg\" (UniqueName: \"kubernetes.io/projected/4f3b3503-699a-4697-b121-351ad81bad8a-kube-api-access-qwfbg\") pod \"4f3b3503-699a-4697-b121-351ad81bad8a\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") "
Nov 03 08:21:22.894163 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.894136    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-istio-data\") pod \"4f3b3503-699a-4697-b121-351ad81bad8a\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") "
Nov 03 08:21:22.894286 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.894247    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f3b3503-699a-4697-b121-351ad81bad8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3b3503-699a-4697-b121-351ad81bad8a-istio-podinfo\") pod \"4f3b3503-699a-4697-b121-351ad81bad8a\" (UID: \"4f3b3503-699a-4697-b121-351ad81bad8a\") "
Nov 03 08:21:22.894357 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.894334    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f3b3503-699a-4697-b121-351ad81bad8a-istiod-ca-cert" pod "4f3b3503-699a-4697-b121-351ad81bad8a" (UID: "4f3b3503-699a-4697-b121-351ad81bad8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:21:22.894486 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.894465    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzt4m\" (UniqueName: \"kubernetes.io/projected/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-kube-api-access-dzt4m\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:22.894882 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.894544    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-token\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:22.894882 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.894545    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-workload-socket" pod "4f3b3503-699a-4697-b121-351ad81bad8a" (UID: "4f3b3503-699a-4697-b121-351ad81bad8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:22.894882 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.894577    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-podinfo\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:22.894882 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.894673    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-data\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:22.894882 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.894694    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istiod-ca-cert\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:22.894882 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.894750    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-envoy\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:22.895185 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.894966    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-workload-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:22.895185 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.894992    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-credential-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:22.895185 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.895014    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-workload-certs\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:22.895185 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.895089    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-workload-certs" pod "4f3b3503-699a-4697-b121-351ad81bad8a" (UID: "4f3b3503-699a-4697-b121-351ad81bad8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:22.895185 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.895158    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f3b3503-699a-4697-b121-351ad81bad8a-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:22.895185 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.895173    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:22.896148 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.896119    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-credential-socket" pod "4f3b3503-699a-4697-b121-351ad81bad8a" (UID: "4f3b3503-699a-4697-b121-351ad81bad8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:22.896264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.896193    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-istio-data" pod "4f3b3503-699a-4697-b121-351ad81bad8a" (UID: "4f3b3503-699a-4697-b121-351ad81bad8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:22.897587 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.897556    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-istio-envoy" pod "4f3b3503-699a-4697-b121-351ad81bad8a" (UID: "4f3b3503-699a-4697-b121-351ad81bad8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:22.898318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.898297    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f3b3503-699a-4697-b121-351ad81bad8a-istio-podinfo" pod "4f3b3503-699a-4697-b121-351ad81bad8a" (UID: "4f3b3503-699a-4697-b121-351ad81bad8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:21:22.898404 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.898382    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f3b3503-699a-4697-b121-351ad81bad8a-kube-api-access-qwfbg" pod "4f3b3503-699a-4697-b121-351ad81bad8a" (UID: "4f3b3503-699a-4697-b121-351ad81bad8a"). InnerVolumeSpecName "kube-api-access-qwfbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:21:22.899731 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.899708    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f3b3503-699a-4697-b121-351ad81bad8a-istio-token" pod "4f3b3503-699a-4697-b121-351ad81bad8a" (UID: "4f3b3503-699a-4697-b121-351ad81bad8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:21:22.900376 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.900356    5070 scope.go:122] "RemoveContainer" containerID="776e6bbde0a6b87bf9d2c28e2d8a298aeabc01552cdab214a0425efce769f8f3"
Nov 03 08:21:22.949096 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.947896    5070 scope.go:122] "RemoveContainer" containerID="41f9ed86772f153450c891f4cf3bbbd6ca5a5c633806e8d922f1621e3a16c7ef"
Nov 03 08:21:22.949096 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:21:22.948216    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f9ed86772f153450c891f4cf3bbbd6ca5a5c633806e8d922f1621e3a16c7ef\": container with ID starting with 41f9ed86772f153450c891f4cf3bbbd6ca5a5c633806e8d922f1621e3a16c7ef not found: ID does not exist" containerID="41f9ed86772f153450c891f4cf3bbbd6ca5a5c633806e8d922f1621e3a16c7ef"
Nov 03 08:21:22.949096 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.948242    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f9ed86772f153450c891f4cf3bbbd6ca5a5c633806e8d922f1621e3a16c7ef"} err="failed to get container status \"41f9ed86772f153450c891f4cf3bbbd6ca5a5c633806e8d922f1621e3a16c7ef\": rpc error: code = NotFound desc = could not find container \"41f9ed86772f153450c891f4cf3bbbd6ca5a5c633806e8d922f1621e3a16c7ef\": container with ID starting with 41f9ed86772f153450c891f4cf3bbbd6ca5a5c633806e8d922f1621e3a16c7ef not found: ID does not exist"
Nov 03 08:21:22.949096 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.948260    5070 scope.go:122] "RemoveContainer" containerID="776e6bbde0a6b87bf9d2c28e2d8a298aeabc01552cdab214a0425efce769f8f3"
Nov 03 08:21:22.949096 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:21:22.948507    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"776e6bbde0a6b87bf9d2c28e2d8a298aeabc01552cdab214a0425efce769f8f3\": container with ID starting with 776e6bbde0a6b87bf9d2c28e2d8a298aeabc01552cdab214a0425efce769f8f3 not found: ID does not exist" containerID="776e6bbde0a6b87bf9d2c28e2d8a298aeabc01552cdab214a0425efce769f8f3"
Nov 03 08:21:22.949096 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.948528    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"776e6bbde0a6b87bf9d2c28e2d8a298aeabc01552cdab214a0425efce769f8f3"} err="failed to get container status \"776e6bbde0a6b87bf9d2c28e2d8a298aeabc01552cdab214a0425efce769f8f3\": rpc error: code = NotFound desc = could not find container \"776e6bbde0a6b87bf9d2c28e2d8a298aeabc01552cdab214a0425efce769f8f3\": container with ID starting with 776e6bbde0a6b87bf9d2c28e2d8a298aeabc01552cdab214a0425efce769f8f3 not found: ID does not exist"
Nov 03 08:21:22.997248 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.997211    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzt4m\" (UniqueName: \"kubernetes.io/projected/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-kube-api-access-dzt4m\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:22.997438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.997298    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-token\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:22.997438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.997334    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-podinfo\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:22.997626 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.997431    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-data\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:22.997626 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.997534    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istiod-ca-cert\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:22.997626 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.997604    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-envoy\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:22.997880 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.997847    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-workload-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:22.997943 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.997887    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-credential-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:22.997943 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.997928    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-workload-certs\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:22.998821 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.998200    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-data\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:22.998821 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.998795    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:22.998821 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.998821    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f3b3503-699a-4697-b121-351ad81bad8a-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:22.999072 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.998838    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f3b3503-699a-4697-b121-351ad81bad8a-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:22.999072 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.998855    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:22.999072 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.998870    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:22.999072 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.998895    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f3b3503-699a-4697-b121-351ad81bad8a-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:22.999072 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.998911    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwfbg\" (UniqueName: \"kubernetes.io/projected/4f3b3503-699a-4697-b121-351ad81bad8a-kube-api-access-qwfbg\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:22.999072 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.998795    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-workload-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:23.000264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.999112    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istiod-ca-cert\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:23.000264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.999405    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-credential-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:23.000264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:22.999500    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-workload-certs\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:23.000435 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:23.000344    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-podinfo\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:23.001724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:23.001700    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-envoy\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:23.009894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:23.009837    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzt4m\" (UniqueName: \"kubernetes.io/projected/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-kube-api-access-dzt4m\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:23.012050 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:23.012026    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-token\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:23.189826 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:23.174054    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:23.263132 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:23.223002    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"]
Nov 03 08:21:23.263132 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:23.250572    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-g862p"]
Nov 03 08:21:23.378236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:23.378037    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"]
Nov 03 08:21:23.378964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:23.378924    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc" podUID="e33edac1-844e-4caf-bb59-d63291659555" containerName="istio-proxy" containerID="cri-o://7d1c88516aa76e6bfff3cf5706a8fcfa0cd75b58781094bcad099e5871bedc1b" gracePeriod=30
Nov 03 08:21:23.716509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:23.716473    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"]
Nov 03 08:21:23.726151 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:21:23.726122    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75f2a3f_2d62_4f7f_a449_eba6bb6cfceb.slice/crio-3c34707ae33c4775ff509e7a7059aa91f0cd1eb3e130b7f863836f1e86a05ca8 WatchSource:0}: Error finding container 3c34707ae33c4775ff509e7a7059aa91f0cd1eb3e130b7f863836f1e86a05ca8: Status 404 returned error can't find the container with id 3c34707ae33c4775ff509e7a7059aa91f0cd1eb3e130b7f863836f1e86a05ca8
Nov 03 08:21:23.730550 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:23.730509    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:21:23.730662 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:23.730595    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:21:23.730662 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:23.730643    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:21:23.948858 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:23.948391    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz" event={"ID":"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb","Type":"ContainerStarted","Data":"e222ed448d5eb626b27ddaf01aa7c0996e4e51c20eed16e3fa24ac4c005a5768"}
Nov 03 08:21:23.948858 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:23.948463    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz" event={"ID":"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb","Type":"ContainerStarted","Data":"3c34707ae33c4775ff509e7a7059aa91f0cd1eb3e130b7f863836f1e86a05ca8"}
Nov 03 08:21:24.014838 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:24.014192    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz" podStartSLOduration=154.014170377 podStartE2EDuration="2m34.014170377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:21:23.987240906 +0000 UTC m=+1682.101704841" watchObservedRunningTime="2026-11-03 08:21:24.014170377 +0000 UTC m=+1682.128634309"
Nov 03 08:21:24.105802 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:24.104116    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f3b3503-699a-4697-b121-351ad81bad8a" path="/var/lib/kubelet/pods/4f3b3503-699a-4697-b121-351ad81bad8a/volumes"
Nov 03 08:21:24.186677 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:24.186644    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:24.188076 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:24.188045    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 08:21:24.188218 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:24.188094    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz" podUID="d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 08:21:25.187382 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:25.187335    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 08:21:25.187892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:25.187402    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz" podUID="d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 08:21:26.187399 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:26.187366    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 08:21:26.187809 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:26.187417    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz" podUID="d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 08:21:27.192372 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:27.192332    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 08:21:27.193042 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:27.193007    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz" podUID="d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 08:21:27.987598 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:27.987561    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk"]
Nov 03 08:21:28.006100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.006064    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk"
Nov 03 08:21:28.010397 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.010367    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0d168e-dockercfg-8mpcb\""
Nov 03 08:21:28.010752 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.010731    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0d168e\""
Nov 03 08:21:28.013679 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.013657    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk"]
Nov 03 08:21:28.047879 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.046430    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t"]
Nov 03 08:21:28.054576 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.054549    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t"]
Nov 03 08:21:28.054740 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.054640    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t"
Nov 03 08:21:28.179780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.155780    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fcd6530-2913-45a2-a36b-697f56d42909-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk\" (UID: \"4fcd6530-2913-45a2-a36b-697f56d42909\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk"
Nov 03 08:21:28.179780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.155978    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phzbm\" (UniqueName: \"kubernetes.io/projected/216f610f-a3c5-4967-949d-fbabca17ed1a-kube-api-access-phzbm\") pod \"payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t\" (UID: \"216f610f-a3c5-4967-949d-fbabca17ed1a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t"
Nov 03 08:21:28.179780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.156117    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/216f610f-a3c5-4967-949d-fbabca17ed1a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t\" (UID: \"216f610f-a3c5-4967-949d-fbabca17ed1a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t"
Nov 03 08:21:28.179780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.156185    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhbmq\" (UniqueName: \"kubernetes.io/projected/4fcd6530-2913-45a2-a36b-697f56d42909-kube-api-access-hhbmq\") pod \"payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk\" (UID: \"4fcd6530-2913-45a2-a36b-697f56d42909\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk"
Nov 03 08:21:28.228934 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.212019    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 08:21:28.228934 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.212097    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz" podUID="d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 08:21:28.436920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.269212    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phzbm\" (UniqueName: \"kubernetes.io/projected/216f610f-a3c5-4967-949d-fbabca17ed1a-kube-api-access-phzbm\") pod \"payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t\" (UID: \"216f610f-a3c5-4967-949d-fbabca17ed1a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t"
Nov 03 08:21:28.436920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.269381    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/216f610f-a3c5-4967-949d-fbabca17ed1a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t\" (UID: \"216f610f-a3c5-4967-949d-fbabca17ed1a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t"
Nov 03 08:21:28.436920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.269450    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhbmq\" (UniqueName: \"kubernetes.io/projected/4fcd6530-2913-45a2-a36b-697f56d42909-kube-api-access-hhbmq\") pod \"payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk\" (UID: \"4fcd6530-2913-45a2-a36b-697f56d42909\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk"
Nov 03 08:21:28.436920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.269685    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fcd6530-2913-45a2-a36b-697f56d42909-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk\" (UID: \"4fcd6530-2913-45a2-a36b-697f56d42909\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk"
Nov 03 08:21:28.436920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.270664    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fcd6530-2913-45a2-a36b-697f56d42909-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk\" (UID: \"4fcd6530-2913-45a2-a36b-697f56d42909\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk"
Nov 03 08:21:28.540932 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.441801    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/216f610f-a3c5-4967-949d-fbabca17ed1a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t\" (UID: \"216f610f-a3c5-4967-949d-fbabca17ed1a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t"
Nov 03 08:21:28.717844 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.715796    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phzbm\" (UniqueName: \"kubernetes.io/projected/216f610f-a3c5-4967-949d-fbabca17ed1a-kube-api-access-phzbm\") pod \"payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t\" (UID: \"216f610f-a3c5-4967-949d-fbabca17ed1a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t"
Nov 03 08:21:28.759683 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.759609    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhbmq\" (UniqueName: \"kubernetes.io/projected/4fcd6530-2913-45a2-a36b-697f56d42909-kube-api-access-hhbmq\") pod \"payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk\" (UID: \"4fcd6530-2913-45a2-a36b-697f56d42909\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk"
Nov 03 08:21:28.836175 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.836140    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t"
Nov 03 08:21:28.930492 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:28.930452    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk"
Nov 03 08:21:29.046042 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.045652    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:21:29.117555 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.117530    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"]
Nov 03 08:21:29.118578 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.118519    5070 cpu_manager.go:414] "containerMap: removing container" podUID="e33edac1-844e-4caf-bb59-d63291659555" containerName="istio-proxy"
Nov 03 08:21:29.118578 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.118544    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e33edac1-844e-4caf-bb59-d63291659555" containerName="istio-proxy"
Nov 03 08:21:29.118863 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.118847    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="e33edac1-844e-4caf-bb59-d63291659555" containerName="istio-proxy"
Nov 03 08:21:29.124516 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.124197    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.152621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.152549    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"]
Nov 03 08:21:29.192720 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.191971    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 08:21:29.192720 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.192039    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz" podUID="d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 08:21:29.201389 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.201362    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e33edac1-844e-4caf-bb59-d63291659555-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33edac1-844e-4caf-bb59-d63291659555-istio-podinfo\") pod \"e33edac1-844e-4caf-bb59-d63291659555\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") "
Nov 03 08:21:29.203480 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.203453    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e33edac1-844e-4caf-bb59-d63291659555-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33edac1-844e-4caf-bb59-d63291659555-istiod-ca-cert\") pod \"e33edac1-844e-4caf-bb59-d63291659555\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") "
Nov 03 08:21:29.203617 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.203546    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-workload-socket\") pod \"e33edac1-844e-4caf-bb59-d63291659555\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") "
Nov 03 08:21:29.203617 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.203609    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-workload-certs\") pod \"e33edac1-844e-4caf-bb59-d63291659555\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") "
Nov 03 08:21:29.203870 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.203645    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e33edac1-844e-4caf-bb59-d63291659555-istio-token\" (UniqueName: \"kubernetes.io/projected/e33edac1-844e-4caf-bb59-d63291659555-istio-token\") pod \"e33edac1-844e-4caf-bb59-d63291659555\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") "
Nov 03 08:21:29.203870 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.203726    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-credential-socket\") pod \"e33edac1-844e-4caf-bb59-d63291659555\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") "
Nov 03 08:21:29.203870 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.203782    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t"]
Nov 03 08:21:29.204089 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.204058    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-istio-envoy\") pod \"e33edac1-844e-4caf-bb59-d63291659555\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") "
Nov 03 08:21:29.204146 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.204117    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-istio-data\") pod \"e33edac1-844e-4caf-bb59-d63291659555\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") "
Nov 03 08:21:29.204212 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.204173    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e33edac1-844e-4caf-bb59-d63291659555-kube-api-access-r6qgv\" (UniqueName: \"kubernetes.io/projected/e33edac1-844e-4caf-bb59-d63291659555-kube-api-access-r6qgv\") pod \"e33edac1-844e-4caf-bb59-d63291659555\" (UID: \"e33edac1-844e-4caf-bb59-d63291659555\") "
Nov 03 08:21:29.204212 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.204190    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-workload-socket" pod "e33edac1-844e-4caf-bb59-d63291659555" (UID: "e33edac1-844e-4caf-bb59-d63291659555"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:29.204516 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.204481    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-workload-certs\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.204586 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.204548    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e33edac1-844e-4caf-bb59-d63291659555-istiod-ca-cert" pod "e33edac1-844e-4caf-bb59-d63291659555" (UID: "e33edac1-844e-4caf-bb59-d63291659555"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:21:29.204706 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.204684    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-podinfo\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.204907 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.204884    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e33edac1-844e-4caf-bb59-d63291659555-istio-podinfo" pod "e33edac1-844e-4caf-bb59-d63291659555" (UID: "e33edac1-844e-4caf-bb59-d63291659555"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:21:29.205122 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.205098    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-envoy\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.205182 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.205165    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-token\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.205247 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.205215    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-credential-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.205302 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.205281    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-workload-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.205369 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.205304    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-data\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.205479 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.205460    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5zsj\" (UniqueName: \"kubernetes.io/projected/825e9af0-37a8-4d27-9a34-20b2f87e92d5-kube-api-access-c5zsj\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.205548 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.205499    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istiod-ca-cert\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.205632 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.205608    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-workload-certs" pod "e33edac1-844e-4caf-bb59-d63291659555" (UID: "e33edac1-844e-4caf-bb59-d63291659555"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:29.206249 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.206197    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e33edac1-844e-4caf-bb59-d63291659555-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:29.206500 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.206479    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e33edac1-844e-4caf-bb59-d63291659555-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:29.206829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.206504    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:29.206829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.206550    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-istio-data" pod "e33edac1-844e-4caf-bb59-d63291659555" (UID: "e33edac1-844e-4caf-bb59-d63291659555"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:29.207150 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.207121    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-credential-socket" pod "e33edac1-844e-4caf-bb59-d63291659555" (UID: "e33edac1-844e-4caf-bb59-d63291659555"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:29.210248 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.210214    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-istio-envoy" pod "e33edac1-844e-4caf-bb59-d63291659555" (UID: "e33edac1-844e-4caf-bb59-d63291659555"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:21:29.261035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.260944    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33edac1-844e-4caf-bb59-d63291659555-kube-api-access-r6qgv" pod "e33edac1-844e-4caf-bb59-d63291659555" (UID: "e33edac1-844e-4caf-bb59-d63291659555"). InnerVolumeSpecName "kube-api-access-r6qgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:21:29.265726 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.265680    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e33edac1-844e-4caf-bb59-d63291659555-istio-token" pod "e33edac1-844e-4caf-bb59-d63291659555" (UID: "e33edac1-844e-4caf-bb59-d63291659555"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:21:29.310621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.308276    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-envoy\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.310621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.308330    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-token\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.310621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.308363    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-credential-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.310621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.308411    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-workload-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.310621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.308429    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-data\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.310621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.308505    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zsj\" (UniqueName: \"kubernetes.io/projected/825e9af0-37a8-4d27-9a34-20b2f87e92d5-kube-api-access-c5zsj\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.310621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.308527    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istiod-ca-cert\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.310621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.308591    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-workload-certs\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.310621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.308668    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-podinfo\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.310621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.308810    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:29.310621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.308821    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e33edac1-844e-4caf-bb59-d63291659555-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:29.310621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.308831    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:29.310621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.308841    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:29.310621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.308850    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e33edac1-844e-4caf-bb59-d63291659555-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:29.310621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.308860    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6qgv\" (UniqueName: \"kubernetes.io/projected/e33edac1-844e-4caf-bb59-d63291659555-kube-api-access-r6qgv\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:29.310621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.310166    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-workload-certs\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.310621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.310242    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-data\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.310621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.310431    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-credential-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.310621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.310531    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istiod-ca-cert\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.310621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.310552    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-workload-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.321386 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.321355    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-podinfo\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.323212 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.323189    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-envoy\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.326182 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.326156    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5zsj\" (UniqueName: \"kubernetes.io/projected/825e9af0-37a8-4d27-9a34-20b2f87e92d5-kube-api-access-c5zsj\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.326358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.326339    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-token\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.495216 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.494231    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:29.507523 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.507478    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk"]
Nov 03 08:21:29.647272 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.646959    5070 generic.go:356] "Generic (PLEG): container finished" podID="e33edac1-844e-4caf-bb59-d63291659555" containerID="7d1c88516aa76e6bfff3cf5706a8fcfa0cd75b58781094bcad099e5871bedc1b" exitCode=0
Nov 03 08:21:29.647272 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.647018    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"
Nov 03 08:21:29.647272 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.647064    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc" event={"ID":"e33edac1-844e-4caf-bb59-d63291659555","Type":"ContainerDied","Data":"7d1c88516aa76e6bfff3cf5706a8fcfa0cd75b58781094bcad099e5871bedc1b"}
Nov 03 08:21:29.647272 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.647110    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc" event={"ID":"e33edac1-844e-4caf-bb59-d63291659555","Type":"ContainerDied","Data":"d0ef283949ee7c95f36a8ad159811c81e199eb0cea967d6a262edf410e12edbc"}
Nov 03 08:21:29.647272 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.647156    5070 scope.go:122] "RemoveContainer" containerID="7d1c88516aa76e6bfff3cf5706a8fcfa0cd75b58781094bcad099e5871bedc1b"
Nov 03 08:21:29.652577 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.652497    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk" event={"ID":"4fcd6530-2913-45a2-a36b-697f56d42909","Type":"ContainerStarted","Data":"0dd54bb791cc1de06fdfcd9bbea955d0359adba87288b618f9e136d60680c0b7"}
Nov 03 08:21:29.659498 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.659404    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t" event={"ID":"216f610f-a3c5-4967-949d-fbabca17ed1a","Type":"ContainerStarted","Data":"b6361dc0082e8ba4e47b63f2a172f96825a618ffec32652a95f141a96a591727"}
Nov 03 08:21:29.659498 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.659459    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t" event={"ID":"216f610f-a3c5-4967-949d-fbabca17ed1a","Type":"ContainerStarted","Data":"0bac3a35831b2fb6c9e83dd48429f4356096871a4ec9a9ba53070ab631971b1d"}
Nov 03 08:21:29.659498 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.659493    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t"
Nov 03 08:21:29.701551 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.701504    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t" podStartSLOduration=1.701488474 podStartE2EDuration="1.701488474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:21:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:21:29.69573599 +0000 UTC m=+1687.810199924" watchObservedRunningTime="2026-11-03 08:21:29.701488474 +0000 UTC m=+1687.815952414"
Nov 03 08:21:29.702412 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.702244    5070 scope.go:122] "RemoveContainer" containerID="7d1c88516aa76e6bfff3cf5706a8fcfa0cd75b58781094bcad099e5871bedc1b"
Nov 03 08:21:29.703554 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:21:29.703525    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d1c88516aa76e6bfff3cf5706a8fcfa0cd75b58781094bcad099e5871bedc1b\": container with ID starting with 7d1c88516aa76e6bfff3cf5706a8fcfa0cd75b58781094bcad099e5871bedc1b not found: ID does not exist" containerID="7d1c88516aa76e6bfff3cf5706a8fcfa0cd75b58781094bcad099e5871bedc1b"
Nov 03 08:21:29.703639 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.703566    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d1c88516aa76e6bfff3cf5706a8fcfa0cd75b58781094bcad099e5871bedc1b"} err="failed to get container status \"7d1c88516aa76e6bfff3cf5706a8fcfa0cd75b58781094bcad099e5871bedc1b\": rpc error: code = NotFound desc = could not find container \"7d1c88516aa76e6bfff3cf5706a8fcfa0cd75b58781094bcad099e5871bedc1b\": container with ID starting with 7d1c88516aa76e6bfff3cf5706a8fcfa0cd75b58781094bcad099e5871bedc1b not found: ID does not exist"
Nov 03 08:21:29.722790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.718970    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"]
Nov 03 08:21:29.725651 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.725630    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-l6bvc"]
Nov 03 08:21:29.911807 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.907921    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"]
Nov 03 08:21:29.936655 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.936615    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:21:29.936833 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.936714    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:21:29.936833 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:29.936787    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:21:30.041018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:30.040972    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e33edac1-844e-4caf-bb59-d63291659555" path="/var/lib/kubelet/pods/e33edac1-844e-4caf-bb59-d63291659555/volumes"
Nov 03 08:21:30.195846 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:30.195002    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 08:21:30.195846 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:30.195087    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz" podUID="d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 08:21:30.705897 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:30.704866    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b" event={"ID":"825e9af0-37a8-4d27-9a34-20b2f87e92d5","Type":"ContainerStarted","Data":"52967583fe3b748836a41c5cc2f924344cc62b23ae21445430993d8895d83eb2"}
Nov 03 08:21:30.705897 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:30.704924    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b" event={"ID":"825e9af0-37a8-4d27-9a34-20b2f87e92d5","Type":"ContainerStarted","Data":"db366a071ef62b89c30772494f3a3cb0e69b55462adbcdd430e311a607d98b33"}
Nov 03 08:21:30.711851 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:30.711744    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk" event={"ID":"4fcd6530-2913-45a2-a36b-697f56d42909","Type":"ContainerStarted","Data":"7f5cb43858694e81fd3723a628ef723d32fff0f13de95c05b81def5ae0a0203e"}
Nov 03 08:21:30.742204 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:30.740691    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b" podStartSLOduration=206.74067226 podStartE2EDuration="3m26.74067226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:21:30.730070852 +0000 UTC m=+1688.844534784" watchObservedRunningTime="2026-11-03 08:21:30.74067226 +0000 UTC m=+1688.855136209"
Nov 03 08:21:30.792044 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:30.791992    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk" podStartSLOduration=3.791978388 podStartE2EDuration="3.791978388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:21:30.771483329 +0000 UTC m=+1688.885947261" watchObservedRunningTime="2026-11-03 08:21:30.791978388 +0000 UTC m=+1688.906442319"
Nov 03 08:21:31.189441 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:31.189404    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Nov 03 08:21:31.189739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:31.189706    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz" podUID="d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Nov 03 08:21:31.502381 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:31.502292    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:31.506366 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:31.506328    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Nov 03 08:21:31.506812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:31.506777    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b" podUID="825e9af0-37a8-4d27-9a34-20b2f87e92d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Nov 03 08:21:31.725281 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:31.725238    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk"
Nov 03 08:21:32.191437 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:32.191405    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:32.191861 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:32.191823    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:32.192321 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:32.192302    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:21:32.504830 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:32.498145    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Nov 03 08:21:32.504830 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:32.498206    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b" podUID="825e9af0-37a8-4d27-9a34-20b2f87e92d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Nov 03 08:21:32.635990 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:32.635953    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e620a55-vb8sh"]
Nov 03 08:21:32.646505 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:32.642839    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e620a55-vb8sh"
Nov 03 08:21:32.652230 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:32.652202    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:21:32.652230 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:32.652219    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:21:32.652432 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:32.652219    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:21:32.652432 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:32.652277    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:21:32.676904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:32.656028    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e620a55-vb8sh"]
Nov 03 08:21:32.690912 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:32.690888    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e620a55-vb8sh\" (UID: \"401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e620a55-vb8sh"
Nov 03 08:21:32.691337 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:32.691314    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvd8h\" (UniqueName: \"kubernetes.io/projected/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7-kube-api-access-cvd8h\") pod \"maas-api-revoke-sub-8e620a55-vb8sh\" (UID: \"401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e620a55-vb8sh"
Nov 03 08:21:32.691584 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:32.691565    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e620a55-vb8sh\" (UID: \"401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e620a55-vb8sh"
Nov 03 08:21:32.793458 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:32.793366    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e620a55-vb8sh\" (UID: \"401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e620a55-vb8sh"
Nov 03 08:21:32.794077 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:32.794050    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvd8h\" (UniqueName: \"kubernetes.io/projected/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7-kube-api-access-cvd8h\") pod \"maas-api-revoke-sub-8e620a55-vb8sh\" (UID: \"401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e620a55-vb8sh"
Nov 03 08:21:32.794405 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:32.794383    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e620a55-vb8sh\" (UID: \"401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e620a55-vb8sh"
Nov 03 08:21:32.794546 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:32.794473    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e620a55-vb8sh\" (UID: \"401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e620a55-vb8sh"
Nov 03 08:21:32.796961 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:32.796923    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e620a55-vb8sh\" (UID: \"401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e620a55-vb8sh"
Nov 03 08:21:32.801504 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:32.801477    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvd8h\" (UniqueName: \"kubernetes.io/projected/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7-kube-api-access-cvd8h\") pod \"maas-api-revoke-sub-8e620a55-vb8sh\" (UID: \"401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e620a55-vb8sh"
Nov 03 08:21:33.000497 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:33.000466    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e620a55-vb8sh"
Nov 03 08:21:33.311749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:33.311716    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e620a55-vb8sh"]
Nov 03 08:21:33.315874 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:21:33.315844    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401e341b_69d7_4c3f_ad3c_9dcc5b1dc8b7.slice/crio-3d9900ce7d58a2ee76888c3560fd68d17ac3a8a1e8ed8c1219ab8b28068f945b WatchSource:0}: Error finding container 3d9900ce7d58a2ee76888c3560fd68d17ac3a8a1e8ed8c1219ab8b28068f945b: Status 404 returned error can't find the container with id 3d9900ce7d58a2ee76888c3560fd68d17ac3a8a1e8ed8c1219ab8b28068f945b
Nov 03 08:21:33.498311 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:33.498276    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Nov 03 08:21:33.498628 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:33.498579    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b" podUID="825e9af0-37a8-4d27-9a34-20b2f87e92d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Nov 03 08:21:33.791885 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:33.791841    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e620a55-vb8sh" event={"ID":"401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7","Type":"ContainerStarted","Data":"3d9900ce7d58a2ee76888c3560fd68d17ac3a8a1e8ed8c1219ab8b28068f945b"}
Nov 03 08:21:34.495325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:34.495291    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Nov 03 08:21:34.495714 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:34.495342    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b" podUID="825e9af0-37a8-4d27-9a34-20b2f87e92d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Nov 03 08:21:34.803063 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:34.802980    5070 generic.go:356] "Generic (PLEG): container finished" podID="401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7" containerID="aa953843af2fe6ebf8b22a847ddcb29b528c15a77f75d0c52963da6a8f9b93ab" exitCode=0
Nov 03 08:21:34.803202 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:34.803060    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e620a55-vb8sh" event={"ID":"401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7","Type":"ContainerDied","Data":"aa953843af2fe6ebf8b22a847ddcb29b528c15a77f75d0c52963da6a8f9b93ab"}
Nov 03 08:21:35.495568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:35.495535    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Nov 03 08:21:35.496086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:35.495584    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b" podUID="825e9af0-37a8-4d27-9a34-20b2f87e92d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Nov 03 08:21:36.196454 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:36.196429    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e620a55-vb8sh"
Nov 03 08:21:36.343724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:36.343686    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7-maas-api-openshift-ca\") pod \"401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7\" (UID: \"401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7\") "
Nov 03 08:21:36.343953 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:36.343798    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7-maas-api-xks-ca\") pod \"401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7\" (UID: \"401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7\") "
Nov 03 08:21:36.343953 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:36.343824    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7-kube-api-access-cvd8h\" (UniqueName: \"kubernetes.io/projected/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7-kube-api-access-cvd8h\") pod \"401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7\" (UID: \"401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7\") "
Nov 03 08:21:36.344130 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:36.344074    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7-maas-api-openshift-ca" pod "401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7" (UID: "401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:21:36.344584 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:36.344561    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:36.346258 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:36.346233    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7-maas-api-xks-ca" pod "401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7" (UID: "401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:21:36.346349 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:36.346262    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7-kube-api-access-cvd8h" pod "401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7" (UID: "401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7"). InnerVolumeSpecName "kube-api-access-cvd8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:21:36.446800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:36.446693    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:36.446800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:36.446722    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvd8h\" (UniqueName: \"kubernetes.io/projected/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7-kube-api-access-cvd8h\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:36.495418 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:36.495381    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused" start-of-body=
Nov 03 08:21:36.497010 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:36.495434    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b" podUID="825e9af0-37a8-4d27-9a34-20b2f87e92d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.222:15021/healthz/ready\": dial tcp 10.132.0.222:15021: connect: connection refused"
Nov 03 08:21:36.822018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:36.821937    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e620a55-vb8sh"
Nov 03 08:21:36.822018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:36.821964    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e620a55-vb8sh" event={"ID":"401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7","Type":"ContainerDied","Data":"3d9900ce7d58a2ee76888c3560fd68d17ac3a8a1e8ed8c1219ab8b28068f945b"}
Nov 03 08:21:36.822018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:36.822011    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d9900ce7d58a2ee76888c3560fd68d17ac3a8a1e8ed8c1219ab8b28068f945b"
Nov 03 08:21:37.531864 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:37.531833    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:37.533328 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:37.533237    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:37.534299 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:37.534276    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:21:40.741534 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:40.741497    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t"
Nov 03 08:21:42.260145 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.260113    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k"]
Nov 03 08:21:42.260978 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.260961    5070 cpu_manager.go:414] "containerMap: removing container" podUID="401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7" containerName="revoke-keys"
Nov 03 08:21:42.261035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.260979    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7" containerName="revoke-keys"
Nov 03 08:21:42.261167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.261155    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7" containerName="revoke-keys"
Nov 03 08:21:42.266990 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.266949    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k"
Nov 03 08:21:42.270091 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.270065    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:21:42.270357 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.270340    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:21:42.270509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.270497    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:21:42.273188 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.270836    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:21:42.274562 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.273827    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k"]
Nov 03 08:21:42.408540 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.408496    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5bc4691-05c4-400c-b66e-af57631bf312-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k\" (UID: \"c5bc4691-05c4-400c-b66e-af57631bf312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k"
Nov 03 08:21:42.408796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.408591    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2rlj\" (UniqueName: \"kubernetes.io/projected/c5bc4691-05c4-400c-b66e-af57631bf312-kube-api-access-w2rlj\") pod \"maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k\" (UID: \"c5bc4691-05c4-400c-b66e-af57631bf312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k"
Nov 03 08:21:42.408932 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.408901    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5bc4691-05c4-400c-b66e-af57631bf312-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k\" (UID: \"c5bc4691-05c4-400c-b66e-af57631bf312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k"
Nov 03 08:21:42.510906 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.510816    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5bc4691-05c4-400c-b66e-af57631bf312-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k\" (UID: \"c5bc4691-05c4-400c-b66e-af57631bf312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k"
Nov 03 08:21:42.511192 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.511172    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5bc4691-05c4-400c-b66e-af57631bf312-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k\" (UID: \"c5bc4691-05c4-400c-b66e-af57631bf312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k"
Nov 03 08:21:42.511256 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.511247    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2rlj\" (UniqueName: \"kubernetes.io/projected/c5bc4691-05c4-400c-b66e-af57631bf312-kube-api-access-w2rlj\") pod \"maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k\" (UID: \"c5bc4691-05c4-400c-b66e-af57631bf312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k"
Nov 03 08:21:42.511872 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.511852    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5bc4691-05c4-400c-b66e-af57631bf312-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k\" (UID: \"c5bc4691-05c4-400c-b66e-af57631bf312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k"
Nov 03 08:21:42.513326 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.513302    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5bc4691-05c4-400c-b66e-af57631bf312-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k\" (UID: \"c5bc4691-05c4-400c-b66e-af57631bf312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k"
Nov 03 08:21:42.518099 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.518077    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2rlj\" (UniqueName: \"kubernetes.io/projected/c5bc4691-05c4-400c-b66e-af57631bf312-kube-api-access-w2rlj\") pod \"maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k\" (UID: \"c5bc4691-05c4-400c-b66e-af57631bf312\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k"
Nov 03 08:21:42.654834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.654809    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k"
Nov 03 08:21:42.743814 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.739133    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk"
Nov 03 08:21:42.942023 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:42.941715    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k"]
Nov 03 08:21:43.898613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:43.898525    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k" event={"ID":"c5bc4691-05c4-400c-b66e-af57631bf312","Type":"ContainerStarted","Data":"4e95d18141a4fcbb840da5b6169fce01816da57b01ab9effe9d6e517fe26fe13"}
Nov 03 08:21:43.898613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:43.898585    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k" event={"ID":"c5bc4691-05c4-400c-b66e-af57631bf312","Type":"ContainerStarted","Data":"a3de81ea592b66be54d8019ba3f354bfb8223387209c5e6ff197ec5d23126c00"}
Nov 03 08:21:43.920395 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:43.919957    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k" podStartSLOduration=1.91994088 podStartE2EDuration="1.91994088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:21:43.917837031 +0000 UTC m=+1702.032300976" watchObservedRunningTime="2026-11-03 08:21:43.91994088 +0000 UTC m=+1702.034404812"
Nov 03 08:21:44.990783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:44.985206    5070 generic.go:356] "Generic (PLEG): container finished" podID="c5bc4691-05c4-400c-b66e-af57631bf312" containerID="4e95d18141a4fcbb840da5b6169fce01816da57b01ab9effe9d6e517fe26fe13" exitCode=0
Nov 03 08:21:44.990783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:44.985566    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k" event={"ID":"c5bc4691-05c4-400c-b66e-af57631bf312","Type":"ContainerDied","Data":"4e95d18141a4fcbb840da5b6169fce01816da57b01ab9effe9d6e517fe26fe13"}
Nov 03 08:21:46.299851 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:46.299825    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k"
Nov 03 08:21:46.416050 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:46.416023    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5bc4691-05c4-400c-b66e-af57631bf312-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5bc4691-05c4-400c-b66e-af57631bf312-maas-api-openshift-ca\") pod \"c5bc4691-05c4-400c-b66e-af57631bf312\" (UID: \"c5bc4691-05c4-400c-b66e-af57631bf312\") "
Nov 03 08:21:46.416222 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:46.416179    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5bc4691-05c4-400c-b66e-af57631bf312-kube-api-access-w2rlj\" (UniqueName: \"kubernetes.io/projected/c5bc4691-05c4-400c-b66e-af57631bf312-kube-api-access-w2rlj\") pod \"c5bc4691-05c4-400c-b66e-af57631bf312\" (UID: \"c5bc4691-05c4-400c-b66e-af57631bf312\") "
Nov 03 08:21:46.416295 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:46.416266    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5bc4691-05c4-400c-b66e-af57631bf312-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5bc4691-05c4-400c-b66e-af57631bf312-maas-api-xks-ca\") pod \"c5bc4691-05c4-400c-b66e-af57631bf312\" (UID: \"c5bc4691-05c4-400c-b66e-af57631bf312\") "
Nov 03 08:21:46.416438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:46.416411    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5bc4691-05c4-400c-b66e-af57631bf312-maas-api-openshift-ca" pod "c5bc4691-05c4-400c-b66e-af57631bf312" (UID: "c5bc4691-05c4-400c-b66e-af57631bf312"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:21:46.416916 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:46.416892    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5bc4691-05c4-400c-b66e-af57631bf312-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:46.418359 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:46.418334    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5bc4691-05c4-400c-b66e-af57631bf312-kube-api-access-w2rlj" pod "c5bc4691-05c4-400c-b66e-af57631bf312" (UID: "c5bc4691-05c4-400c-b66e-af57631bf312"). InnerVolumeSpecName "kube-api-access-w2rlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:21:46.418458 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:46.418356    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5bc4691-05c4-400c-b66e-af57631bf312-maas-api-xks-ca" pod "c5bc4691-05c4-400c-b66e-af57631bf312" (UID: "c5bc4691-05c4-400c-b66e-af57631bf312"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:21:46.518811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:46.518721    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2rlj\" (UniqueName: \"kubernetes.io/projected/c5bc4691-05c4-400c-b66e-af57631bf312-kube-api-access-w2rlj\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:46.518811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:46.518745    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5bc4691-05c4-400c-b66e-af57631bf312-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:21:47.004459 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:47.004430    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k"
Nov 03 08:21:47.004632 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:47.004431    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k" event={"ID":"c5bc4691-05c4-400c-b66e-af57631bf312","Type":"ContainerDied","Data":"a3de81ea592b66be54d8019ba3f354bfb8223387209c5e6ff197ec5d23126c00"}
Nov 03 08:21:47.004632 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:47.004559    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3de81ea592b66be54d8019ba3f354bfb8223387209c5e6ff197ec5d23126c00"
Nov 03 08:21:52.056286 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:21:52.056254    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:21:54.111116 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:54.111079    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k"]
Nov 03 08:21:54.120834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:54.120774    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0c8a52-zkx2k"]
Nov 03 08:21:54.838661 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:54.838629    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9"]
Nov 03 08:21:54.857803 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:54.857742    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g"]
Nov 03 08:21:56.027952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:56.027918    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5bc4691-05c4-400c-b66e-af57631bf312" path="/var/lib/kubelet/pods/c5bc4691-05c4-400c-b66e-af57631bf312/volumes"
Nov 03 08:21:59.839660 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:59.839599    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9" podUID="115f98fe-906d-4427-ab88-ab890bd2ab66" containerName="payload-processing" containerID="cri-o://13c498fc2743821864b10fdd308ddfe1b1737fafcf5b82f4a65bfa5667c13279" gracePeriod=25
Nov 03 08:21:59.858916 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:21:59.858885    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g" podUID="71db1c49-1b13-498f-a39f-267b76067aee" containerName="payload-pre-processing" containerID="cri-o://ff17b2580aa1b6ef4db8b026cc4140f293f3eae6f9f660ab842e8a2c281adee2" gracePeriod=25
Nov 03 08:22:00.205813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:00.205588    5070 generic.go:356] "Generic (PLEG): container finished" podID="71db1c49-1b13-498f-a39f-267b76067aee" containerID="ff17b2580aa1b6ef4db8b026cc4140f293f3eae6f9f660ab842e8a2c281adee2" exitCode=0
Nov 03 08:22:00.205813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:00.205716    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g" event={"ID":"71db1c49-1b13-498f-a39f-267b76067aee","Type":"ContainerDied","Data":"ff17b2580aa1b6ef4db8b026cc4140f293f3eae6f9f660ab842e8a2c281adee2"}
Nov 03 08:22:00.208716 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:00.208537    5070 generic.go:356] "Generic (PLEG): container finished" podID="115f98fe-906d-4427-ab88-ab890bd2ab66" containerID="13c498fc2743821864b10fdd308ddfe1b1737fafcf5b82f4a65bfa5667c13279" exitCode=0
Nov 03 08:22:00.208716 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:00.208640    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9" event={"ID":"115f98fe-906d-4427-ab88-ab890bd2ab66","Type":"ContainerDied","Data":"13c498fc2743821864b10fdd308ddfe1b1737fafcf5b82f4a65bfa5667c13279"}
Nov 03 08:22:00.276677 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:00.276637    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9"
Nov 03 08:22:00.331409 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:00.331384    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g"
Nov 03 08:22:00.361100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:00.361073    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/115f98fe-906d-4427-ab88-ab890bd2ab66-kube-api-access-2m9n2\" (UniqueName: \"kubernetes.io/projected/115f98fe-906d-4427-ab88-ab890bd2ab66-kube-api-access-2m9n2\") pod \"115f98fe-906d-4427-ab88-ab890bd2ab66\" (UID: \"115f98fe-906d-4427-ab88-ab890bd2ab66\") "
Nov 03 08:22:00.361327 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:00.361313    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71db1c49-1b13-498f-a39f-267b76067aee-kube-api-access-6tnkx\" (UniqueName: \"kubernetes.io/projected/71db1c49-1b13-498f-a39f-267b76067aee-kube-api-access-6tnkx\") pod \"71db1c49-1b13-498f-a39f-267b76067aee\" (UID: \"71db1c49-1b13-498f-a39f-267b76067aee\") "
Nov 03 08:22:00.361423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:00.361351    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71db1c49-1b13-498f-a39f-267b76067aee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71db1c49-1b13-498f-a39f-267b76067aee-plugins-config-volume\") pod \"71db1c49-1b13-498f-a39f-267b76067aee\" (UID: \"71db1c49-1b13-498f-a39f-267b76067aee\") "
Nov 03 08:22:00.361423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:00.361406    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/115f98fe-906d-4427-ab88-ab890bd2ab66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/115f98fe-906d-4427-ab88-ab890bd2ab66-plugins-config-volume\") pod \"115f98fe-906d-4427-ab88-ab890bd2ab66\" (UID: \"115f98fe-906d-4427-ab88-ab890bd2ab66\") "
Nov 03 08:22:00.361973 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:00.361827    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71db1c49-1b13-498f-a39f-267b76067aee-plugins-config-volume" pod "71db1c49-1b13-498f-a39f-267b76067aee" (UID: "71db1c49-1b13-498f-a39f-267b76067aee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:22:00.362103 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:00.362024    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/115f98fe-906d-4427-ab88-ab890bd2ab66-plugins-config-volume" pod "115f98fe-906d-4427-ab88-ab890bd2ab66" (UID: "115f98fe-906d-4427-ab88-ab890bd2ab66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:22:00.362351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:00.362326    5070 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71db1c49-1b13-498f-a39f-267b76067aee-plugins-config-volume\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:00.362444 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:00.362353    5070 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/115f98fe-906d-4427-ab88-ab890bd2ab66-plugins-config-volume\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:00.363466 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:00.363437    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/115f98fe-906d-4427-ab88-ab890bd2ab66-kube-api-access-2m9n2" pod "115f98fe-906d-4427-ab88-ab890bd2ab66" (UID: "115f98fe-906d-4427-ab88-ab890bd2ab66"). InnerVolumeSpecName "kube-api-access-2m9n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:00.364212 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:00.364190    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71db1c49-1b13-498f-a39f-267b76067aee-kube-api-access-6tnkx" pod "71db1c49-1b13-498f-a39f-267b76067aee" (UID: "71db1c49-1b13-498f-a39f-267b76067aee"). InnerVolumeSpecName "kube-api-access-6tnkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:00.464075 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:00.464012    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tnkx\" (UniqueName: \"kubernetes.io/projected/71db1c49-1b13-498f-a39f-267b76067aee-kube-api-access-6tnkx\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:00.464075 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:00.464038    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m9n2\" (UniqueName: \"kubernetes.io/projected/115f98fe-906d-4427-ab88-ab890bd2ab66-kube-api-access-2m9n2\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:01.217868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:01.217750    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g"
Nov 03 08:22:01.217868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:01.217818    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g" event={"ID":"71db1c49-1b13-498f-a39f-267b76067aee","Type":"ContainerDied","Data":"2b34f8adab1a1b8850e34d4cb74765813034f178b834e37f60d714e7cb6c4c38"}
Nov 03 08:22:01.218350 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:01.217880    5070 scope.go:122] "RemoveContainer" containerID="ff17b2580aa1b6ef4db8b026cc4140f293f3eae6f9f660ab842e8a2c281adee2"
Nov 03 08:22:01.219467 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:01.219443    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9"
Nov 03 08:22:01.219625 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:01.219504    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9" event={"ID":"115f98fe-906d-4427-ab88-ab890bd2ab66","Type":"ContainerDied","Data":"56f80d554748db89a158a1844f9ededc7dae37cd40bb8dfdf7bc7b86bbc0454c"}
Nov 03 08:22:01.250931 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:01.250826    5070 scope.go:122] "RemoveContainer" containerID="13c498fc2743821864b10fdd308ddfe1b1737fafcf5b82f4a65bfa5667c13279"
Nov 03 08:22:01.265540 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:01.265516    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9"]
Nov 03 08:22:01.276365 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:01.276342    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7f6f94-7b9c678f47-d6ll9"]
Nov 03 08:22:01.289820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:01.289799    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g"]
Nov 03 08:22:01.294321 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:01.294300    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7f6f94-7bbb7c6dff-d2c4g"]
Nov 03 08:22:01.713918 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:01.713886    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"]
Nov 03 08:22:01.714262 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:01.714238    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s" podUID="312a9c42-79df-4b2d-b430-0c02d7c37200" containerName="istio-proxy" containerID="cri-o://446b1efcb31edfa74e756f8ff3af675dd6ecbf93e4b559de3376f928b7a2f2d8" gracePeriod=30
Nov 03 08:22:02.141790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:02.140955    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="115f98fe-906d-4427-ab88-ab890bd2ab66" path="/var/lib/kubelet/pods/115f98fe-906d-4427-ab88-ab890bd2ab66/volumes"
Nov 03 08:22:02.173782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:02.170029    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71db1c49-1b13-498f-a39f-267b76067aee" path="/var/lib/kubelet/pods/71db1c49-1b13-498f-a39f-267b76067aee/volumes"
Nov 03 08:22:02.372784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:02.368870    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"]
Nov 03 08:22:02.372784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:02.369125    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7" podUID="a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" containerName="istio-proxy" containerID="cri-o://0fa8ebd3a1da0d6692fcdb1427352c97a6148ae8aefc7656bf0c48b4f692ef15" gracePeriod=30
Nov 03 08:22:02.713196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:02.713160    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"]
Nov 03 08:22:02.713377 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:02.713339    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt" podUID="56d6e143-5bb7-42a8-a5e6-0f14214b7592" containerName="istio-proxy" containerID="cri-o://ac0bcc62511738a8bf8144be94bda9ef0d8664b9bea75a800ef8c4b0d9876744" gracePeriod=30
Nov 03 08:22:03.314480 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.314448    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:22:03.763782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.750477    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf"]
Nov 03 08:22:03.764273 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.764090    5070 cpu_manager.go:414] "containerMap: removing container" podUID="71db1c49-1b13-498f-a39f-267b76067aee" containerName="payload-pre-processing"
Nov 03 08:22:03.764273 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.764117    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71db1c49-1b13-498f-a39f-267b76067aee" containerName="payload-pre-processing"
Nov 03 08:22:03.764273 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.764164    5070 cpu_manager.go:414] "containerMap: removing container" podUID="115f98fe-906d-4427-ab88-ab890bd2ab66" containerName="payload-processing"
Nov 03 08:22:03.764273 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.764172    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="115f98fe-906d-4427-ab88-ab890bd2ab66" containerName="payload-processing"
Nov 03 08:22:03.764273 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.764190    5070 cpu_manager.go:414] "containerMap: removing container" podUID="c5bc4691-05c4-400c-b66e-af57631bf312" containerName="revoke-keys"
Nov 03 08:22:03.764273 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.764197    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5bc4691-05c4-400c-b66e-af57631bf312" containerName="revoke-keys"
Nov 03 08:22:03.768225 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.765011    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="71db1c49-1b13-498f-a39f-267b76067aee" containerName="payload-pre-processing"
Nov 03 08:22:03.768225 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.765070    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="115f98fe-906d-4427-ab88-ab890bd2ab66" containerName="payload-processing"
Nov 03 08:22:03.768225 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.765087    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5bc4691-05c4-400c-b66e-af57631bf312" containerName="revoke-keys"
Nov 03 08:22:03.773179 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.773158    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf"
Nov 03 08:22:03.780018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.779992    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:22:03.780303 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.780277    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:22:03.780448 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.780327    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:22:03.780448 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.780366    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:22:03.796373 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.796319    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf"]
Nov 03 08:22:03.824472 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.824450    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rfxp\" (UniqueName: \"kubernetes.io/projected/4160dd6e-4fae-407d-8573-b2e4d840b9f4-kube-api-access-8rfxp\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf\" (UID: \"4160dd6e-4fae-407d-8573-b2e4d840b9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf"
Nov 03 08:22:03.826232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.826193    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4160dd6e-4fae-407d-8573-b2e4d840b9f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf\" (UID: \"4160dd6e-4fae-407d-8573-b2e4d840b9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf"
Nov 03 08:22:03.826374 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.826351    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4160dd6e-4fae-407d-8573-b2e4d840b9f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf\" (UID: \"4160dd6e-4fae-407d-8573-b2e4d840b9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf"
Nov 03 08:22:03.929340 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.929304    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rfxp\" (UniqueName: \"kubernetes.io/projected/4160dd6e-4fae-407d-8573-b2e4d840b9f4-kube-api-access-8rfxp\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf\" (UID: \"4160dd6e-4fae-407d-8573-b2e4d840b9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf"
Nov 03 08:22:03.929569 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.929545    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4160dd6e-4fae-407d-8573-b2e4d840b9f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf\" (UID: \"4160dd6e-4fae-407d-8573-b2e4d840b9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf"
Nov 03 08:22:03.929692 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.929673    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4160dd6e-4fae-407d-8573-b2e4d840b9f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf\" (UID: \"4160dd6e-4fae-407d-8573-b2e4d840b9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf"
Nov 03 08:22:03.930856 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.930832    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4160dd6e-4fae-407d-8573-b2e4d840b9f4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf\" (UID: \"4160dd6e-4fae-407d-8573-b2e4d840b9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf"
Nov 03 08:22:03.932006 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.931985    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4160dd6e-4fae-407d-8573-b2e4d840b9f4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf\" (UID: \"4160dd6e-4fae-407d-8573-b2e4d840b9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf"
Nov 03 08:22:03.937579 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:03.937549    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rfxp\" (UniqueName: \"kubernetes.io/projected/4160dd6e-4fae-407d-8573-b2e4d840b9f4-kube-api-access-8rfxp\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf\" (UID: \"4160dd6e-4fae-407d-8573-b2e4d840b9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf"
Nov 03 08:22:04.094595 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:04.094514    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf"
Nov 03 08:22:04.497187 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:04.493615    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf"]
Nov 03 08:22:04.679786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:04.679454    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf" event={"ID":"4160dd6e-4fae-407d-8573-b2e4d840b9f4","Type":"ContainerStarted","Data":"926014a817fb1c09fdba169339ececb27f35461a7f48cae51eaba5d4a42021a2"}
Nov 03 08:22:05.252817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:05.245504    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq"]
Nov 03 08:22:05.261897 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:05.258056    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq"
Nov 03 08:22:05.325858 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:05.325816    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq"]
Nov 03 08:22:05.394815 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:05.394728    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6c0b1f1-402f-4eb0-adae-9aa037a900e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq\" (UID: \"f6c0b1f1-402f-4eb0-adae-9aa037a900e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq"
Nov 03 08:22:05.394952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:05.394864    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzrct\" (UniqueName: \"kubernetes.io/projected/f6c0b1f1-402f-4eb0-adae-9aa037a900e5-kube-api-access-vzrct\") pod \"maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq\" (UID: \"f6c0b1f1-402f-4eb0-adae-9aa037a900e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq"
Nov 03 08:22:05.394995 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:05.394980    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6c0b1f1-402f-4eb0-adae-9aa037a900e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq\" (UID: \"f6c0b1f1-402f-4eb0-adae-9aa037a900e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq"
Nov 03 08:22:05.496443 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:05.496405    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6c0b1f1-402f-4eb0-adae-9aa037a900e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq\" (UID: \"f6c0b1f1-402f-4eb0-adae-9aa037a900e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq"
Nov 03 08:22:05.496828 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:05.496807    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6c0b1f1-402f-4eb0-adae-9aa037a900e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq\" (UID: \"f6c0b1f1-402f-4eb0-adae-9aa037a900e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq"
Nov 03 08:22:05.496945 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:05.496920    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzrct\" (UniqueName: \"kubernetes.io/projected/f6c0b1f1-402f-4eb0-adae-9aa037a900e5-kube-api-access-vzrct\") pod \"maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq\" (UID: \"f6c0b1f1-402f-4eb0-adae-9aa037a900e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq"
Nov 03 08:22:05.497589 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:05.497566    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6c0b1f1-402f-4eb0-adae-9aa037a900e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq\" (UID: \"f6c0b1f1-402f-4eb0-adae-9aa037a900e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq"
Nov 03 08:22:05.499026 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:05.499005    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6c0b1f1-402f-4eb0-adae-9aa037a900e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq\" (UID: \"f6c0b1f1-402f-4eb0-adae-9aa037a900e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq"
Nov 03 08:22:05.504999 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:05.504963    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzrct\" (UniqueName: \"kubernetes.io/projected/f6c0b1f1-402f-4eb0-adae-9aa037a900e5-kube-api-access-vzrct\") pod \"maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq\" (UID: \"f6c0b1f1-402f-4eb0-adae-9aa037a900e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq"
Nov 03 08:22:05.681912 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:05.681825    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq"
Nov 03 08:22:05.695182 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:05.695150    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf" event={"ID":"4160dd6e-4fae-407d-8573-b2e4d840b9f4","Type":"ContainerStarted","Data":"b229960f18c29960c64d79f8cc477319882eed5389c002ebcaf0e0c8ca6d66e0"}
Nov 03 08:22:05.710537 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:05.710482    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf" podStartSLOduration=2.710463877 podStartE2EDuration="2.710463877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:22:05.709367824 +0000 UTC m=+1723.823831757" watchObservedRunningTime="2026-11-03 08:22:05.710463877 +0000 UTC m=+1723.824927821"
Nov 03 08:22:06.011162 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:06.011126    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq"]
Nov 03 08:22:06.038286 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:22:06.038248    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c0b1f1_402f_4eb0_adae_9aa037a900e5.slice/crio-345878d74e9b26319dd185193e1c7795209ad5e030fc6d1702619b3f3f98b8c9 WatchSource:0}: Error finding container 345878d74e9b26319dd185193e1c7795209ad5e030fc6d1702619b3f3f98b8c9: Status 404 returned error can't find the container with id 345878d74e9b26319dd185193e1c7795209ad5e030fc6d1702619b3f3f98b8c9
Nov 03 08:22:06.705856 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:06.705772    5070 generic.go:356] "Generic (PLEG): container finished" podID="4160dd6e-4fae-407d-8573-b2e4d840b9f4" containerID="b229960f18c29960c64d79f8cc477319882eed5389c002ebcaf0e0c8ca6d66e0" exitCode=0
Nov 03 08:22:06.706367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:06.705850    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf" event={"ID":"4160dd6e-4fae-407d-8573-b2e4d840b9f4","Type":"ContainerDied","Data":"b229960f18c29960c64d79f8cc477319882eed5389c002ebcaf0e0c8ca6d66e0"}
Nov 03 08:22:06.707719 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:06.707696    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq" event={"ID":"f6c0b1f1-402f-4eb0-adae-9aa037a900e5","Type":"ContainerStarted","Data":"4692f1ed69e37cbd8044180ef3ed595a10bbb7b09705fbaff707a06c14230bef"}
Nov 03 08:22:06.707869 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:06.707739    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq" event={"ID":"f6c0b1f1-402f-4eb0-adae-9aa037a900e5","Type":"ContainerStarted","Data":"345878d74e9b26319dd185193e1c7795209ad5e030fc6d1702619b3f3f98b8c9"}
Nov 03 08:22:06.855624 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:06.855550    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq" podStartSLOduration=1.855525631 podStartE2EDuration="1.855525631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:22:06.736664071 +0000 UTC m=+1724.851128008" watchObservedRunningTime="2026-11-03 08:22:06.855525631 +0000 UTC m=+1724.969989564"
Nov 03 08:22:07.203803 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.203777    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:22:07.237737 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.237705    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"]
Nov 03 08:22:07.238549 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.238526    5070 cpu_manager.go:414] "containerMap: removing container" podUID="312a9c42-79df-4b2d-b430-0c02d7c37200" containerName="istio-proxy"
Nov 03 08:22:07.238549 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.238546    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="312a9c42-79df-4b2d-b430-0c02d7c37200" containerName="istio-proxy"
Nov 03 08:22:07.238801 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.238730    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="312a9c42-79df-4b2d-b430-0c02d7c37200" containerName="istio-proxy"
Nov 03 08:22:07.238801 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.238746    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="312a9c42-79df-4b2d-b430-0c02d7c37200" containerName="istio-proxy"
Nov 03 08:22:07.242843 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.242824    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.247933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.247906    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-credential-socket\") pod \"312a9c42-79df-4b2d-b430-0c02d7c37200\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") "
Nov 03 08:22:07.248051 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.247955    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/312a9c42-79df-4b2d-b430-0c02d7c37200-kube-api-access-nmkng\" (UniqueName: \"kubernetes.io/projected/312a9c42-79df-4b2d-b430-0c02d7c37200-kube-api-access-nmkng\") pod \"312a9c42-79df-4b2d-b430-0c02d7c37200\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") "
Nov 03 08:22:07.248051 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.248033    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/312a9c42-79df-4b2d-b430-0c02d7c37200-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/312a9c42-79df-4b2d-b430-0c02d7c37200-istiod-ca-cert\") pod \"312a9c42-79df-4b2d-b430-0c02d7c37200\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") "
Nov 03 08:22:07.248179 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.248142    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-token\" (UniqueName: \"kubernetes.io/projected/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-token\") pod \"312a9c42-79df-4b2d-b430-0c02d7c37200\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") "
Nov 03 08:22:07.248241 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.248176    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-workload-certs\") pod \"312a9c42-79df-4b2d-b430-0c02d7c37200\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") "
Nov 03 08:22:07.248297 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.248253    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-workload-socket\") pod \"312a9c42-79df-4b2d-b430-0c02d7c37200\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") "
Nov 03 08:22:07.248450 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.248434    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-data\") pod \"312a9c42-79df-4b2d-b430-0c02d7c37200\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") "
Nov 03 08:22:07.248580 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.248565    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-envoy\") pod \"312a9c42-79df-4b2d-b430-0c02d7c37200\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") "
Nov 03 08:22:07.248645 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.248612    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-podinfo\") pod \"312a9c42-79df-4b2d-b430-0c02d7c37200\" (UID: \"312a9c42-79df-4b2d-b430-0c02d7c37200\") "
Nov 03 08:22:07.252292 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.252264    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-credential-socket" pod "312a9c42-79df-4b2d-b430-0c02d7c37200" (UID: "312a9c42-79df-4b2d-b430-0c02d7c37200"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:07.253113 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.252507    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-workload-socket" pod "312a9c42-79df-4b2d-b430-0c02d7c37200" (UID: "312a9c42-79df-4b2d-b430-0c02d7c37200"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:07.253113 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.252690    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-workload-certs" pod "312a9c42-79df-4b2d-b430-0c02d7c37200" (UID: "312a9c42-79df-4b2d-b430-0c02d7c37200"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:07.253113 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.252701    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-data" pod "312a9c42-79df-4b2d-b430-0c02d7c37200" (UID: "312a9c42-79df-4b2d-b430-0c02d7c37200"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:07.253113 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.253060    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/312a9c42-79df-4b2d-b430-0c02d7c37200-istiod-ca-cert" pod "312a9c42-79df-4b2d-b430-0c02d7c37200" (UID: "312a9c42-79df-4b2d-b430-0c02d7c37200"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:22:07.255487 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.255457    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-envoy" pod "312a9c42-79df-4b2d-b430-0c02d7c37200" (UID: "312a9c42-79df-4b2d-b430-0c02d7c37200"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:07.299098 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.299063    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-podinfo" pod "312a9c42-79df-4b2d-b430-0c02d7c37200" (UID: "312a9c42-79df-4b2d-b430-0c02d7c37200"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:22:07.300571 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.300527    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-token" pod "312a9c42-79df-4b2d-b430-0c02d7c37200" (UID: "312a9c42-79df-4b2d-b430-0c02d7c37200"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:07.301029 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.300930    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"]
Nov 03 08:22:07.304453 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.304208    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/312a9c42-79df-4b2d-b430-0c02d7c37200-kube-api-access-nmkng" pod "312a9c42-79df-4b2d-b430-0c02d7c37200" (UID: "312a9c42-79df-4b2d-b430-0c02d7c37200"). InnerVolumeSpecName "kube-api-access-nmkng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:07.350631 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.350604    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fab335e-1024-4086-92ef-49b496f0abc0-istio-token\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.350769 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.350737    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fab335e-1024-4086-92ef-49b496f0abc0-istiod-ca-cert\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.350867 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.350845    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-workload-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.350922 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.350900    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fab335e-1024-4086-92ef-49b496f0abc0-istio-podinfo\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.350968 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.350929    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-istio-envoy\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.351043 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.351030    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blqml\" (UniqueName: \"kubernetes.io/projected/8fab335e-1024-4086-92ef-49b496f0abc0-kube-api-access-blqml\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.351086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.351059    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-workload-certs\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.351159 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.351141    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-istio-data\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.351212 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.351198    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-credential-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.351435 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.351422    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:07.351478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.351438    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:07.351478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.351448    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:07.351478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.351458    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmkng\" (UniqueName: \"kubernetes.io/projected/312a9c42-79df-4b2d-b430-0c02d7c37200-kube-api-access-nmkng\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:07.351478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.351467    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/312a9c42-79df-4b2d-b430-0c02d7c37200-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:07.351478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.351477    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:07.351693 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.351486    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:07.351693 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.351494    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:07.351693 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.351503    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/312a9c42-79df-4b2d-b430-0c02d7c37200-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:07.452640 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.452615    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fab335e-1024-4086-92ef-49b496f0abc0-istio-token\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.453315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.452704    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fab335e-1024-4086-92ef-49b496f0abc0-istiod-ca-cert\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.453315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.452747    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-workload-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.453315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.452811    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fab335e-1024-4086-92ef-49b496f0abc0-istio-podinfo\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.453315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.452940    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-istio-envoy\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.453315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.453043    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blqml\" (UniqueName: \"kubernetes.io/projected/8fab335e-1024-4086-92ef-49b496f0abc0-kube-api-access-blqml\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.453315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.453085    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-workload-certs\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.453315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.453153    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-istio-data\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.453315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.453174    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-workload-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.453315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.453198    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-credential-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.453729 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.453356    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fab335e-1024-4086-92ef-49b496f0abc0-istiod-ca-cert\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.453729 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.453461    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-workload-certs\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.453729 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.453499    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-istio-data\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.453729 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.453520    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-credential-socket\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.455140 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.455121    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-istio-envoy\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.455390 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.455370    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fab335e-1024-4086-92ef-49b496f0abc0-istio-podinfo\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.483820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.483775    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fab335e-1024-4086-92ef-49b496f0abc0-istio-token\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.483981 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.483960    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blqml\" (UniqueName: \"kubernetes.io/projected/8fab335e-1024-4086-92ef-49b496f0abc0-kube-api-access-blqml\") pod \"e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") " pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.703408 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.703114    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:07.825341 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.824696    5070 generic.go:356] "Generic (PLEG): container finished" podID="f6c0b1f1-402f-4eb0-adae-9aa037a900e5" containerID="4692f1ed69e37cbd8044180ef3ed595a10bbb7b09705fbaff707a06c14230bef" exitCode=0
Nov 03 08:22:07.825341 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.824899    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq" event={"ID":"f6c0b1f1-402f-4eb0-adae-9aa037a900e5","Type":"ContainerDied","Data":"4692f1ed69e37cbd8044180ef3ed595a10bbb7b09705fbaff707a06c14230bef"}
Nov 03 08:22:07.829439 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.829071    5070 generic.go:356] "Generic (PLEG): container finished" podID="a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" containerID="0fa8ebd3a1da0d6692fcdb1427352c97a6148ae8aefc7656bf0c48b4f692ef15" exitCode=0
Nov 03 08:22:07.829439 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.829364    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7" event={"ID":"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4","Type":"ContainerDied","Data":"0fa8ebd3a1da0d6692fcdb1427352c97a6148ae8aefc7656bf0c48b4f692ef15"}
Nov 03 08:22:07.839708 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.839615    5070 generic.go:356] "Generic (PLEG): container finished" podID="312a9c42-79df-4b2d-b430-0c02d7c37200" containerID="446b1efcb31edfa74e756f8ff3af675dd6ecbf93e4b559de3376f928b7a2f2d8" exitCode=0
Nov 03 08:22:07.840095 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.840079    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"
Nov 03 08:22:07.841395 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.841375    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s" event={"ID":"312a9c42-79df-4b2d-b430-0c02d7c37200","Type":"ContainerDied","Data":"446b1efcb31edfa74e756f8ff3af675dd6ecbf93e4b559de3376f928b7a2f2d8"}
Nov 03 08:22:07.842645 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.842623    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s" event={"ID":"312a9c42-79df-4b2d-b430-0c02d7c37200","Type":"ContainerDied","Data":"1a0661c239177ea72b63fbd95b1f755bbdaad3e776d38c229d4bba50955fb381"}
Nov 03 08:22:07.842768 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.842742    5070 scope.go:122] "RemoveContainer" containerID="446b1efcb31edfa74e756f8ff3af675dd6ecbf93e4b559de3376f928b7a2f2d8"
Nov 03 08:22:07.917407 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.917382    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:22:07.918233 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.918186    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"]
Nov 03 08:22:07.923576 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.923495    5070 scope.go:122] "RemoveContainer" containerID="f9a91dc4463a8e8c4deb22688cea1686bf757ff9ef5c83b0d7980ed000b3b2af"
Nov 03 08:22:07.927596 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.927572    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0c8a52-openshift-default-79cb79965-mbl4s"]
Nov 03 08:22:07.954461 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.954343    5070 scope.go:122] "RemoveContainer" containerID="446b1efcb31edfa74e756f8ff3af675dd6ecbf93e4b559de3376f928b7a2f2d8"
Nov 03 08:22:07.954946 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:22:07.954924    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"446b1efcb31edfa74e756f8ff3af675dd6ecbf93e4b559de3376f928b7a2f2d8\": container with ID starting with 446b1efcb31edfa74e756f8ff3af675dd6ecbf93e4b559de3376f928b7a2f2d8 not found: ID does not exist" containerID="446b1efcb31edfa74e756f8ff3af675dd6ecbf93e4b559de3376f928b7a2f2d8"
Nov 03 08:22:07.955074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.954955    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"446b1efcb31edfa74e756f8ff3af675dd6ecbf93e4b559de3376f928b7a2f2d8"} err="failed to get container status \"446b1efcb31edfa74e756f8ff3af675dd6ecbf93e4b559de3376f928b7a2f2d8\": rpc error: code = NotFound desc = could not find container \"446b1efcb31edfa74e756f8ff3af675dd6ecbf93e4b559de3376f928b7a2f2d8\": container with ID starting with 446b1efcb31edfa74e756f8ff3af675dd6ecbf93e4b559de3376f928b7a2f2d8 not found: ID does not exist"
Nov 03 08:22:07.955074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.954976    5070 scope.go:122] "RemoveContainer" containerID="f9a91dc4463a8e8c4deb22688cea1686bf757ff9ef5c83b0d7980ed000b3b2af"
Nov 03 08:22:07.955542 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:22:07.955466    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a91dc4463a8e8c4deb22688cea1686bf757ff9ef5c83b0d7980ed000b3b2af\": container with ID starting with f9a91dc4463a8e8c4deb22688cea1686bf757ff9ef5c83b0d7980ed000b3b2af not found: ID does not exist" containerID="f9a91dc4463a8e8c4deb22688cea1686bf757ff9ef5c83b0d7980ed000b3b2af"
Nov 03 08:22:07.955542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:07.955504    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a91dc4463a8e8c4deb22688cea1686bf757ff9ef5c83b0d7980ed000b3b2af"} err="failed to get container status \"f9a91dc4463a8e8c4deb22688cea1686bf757ff9ef5c83b0d7980ed000b3b2af\": rpc error: code = NotFound desc = could not find container \"f9a91dc4463a8e8c4deb22688cea1686bf757ff9ef5c83b0d7980ed000b3b2af\": container with ID starting with f9a91dc4463a8e8c4deb22688cea1686bf757ff9ef5c83b0d7980ed000b3b2af not found: ID does not exist"
Nov 03 08:22:08.018429 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.016555    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-data\") pod \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") "
Nov 03 08:22:08.018429 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.016663    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istiod-ca-cert\") pod \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") "
Nov 03 08:22:08.018429 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.016689    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-token\" (UniqueName: \"kubernetes.io/projected/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-token\") pod \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") "
Nov 03 08:22:08.018429 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.016795    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-workload-socket\") pod \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") "
Nov 03 08:22:08.018429 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.016850    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-kube-api-access-hb9nr\" (UniqueName: \"kubernetes.io/projected/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-kube-api-access-hb9nr\") pod \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") "
Nov 03 08:22:08.018429 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.016912    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-credential-socket\") pod \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") "
Nov 03 08:22:08.018429 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.017015    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-envoy\") pod \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") "
Nov 03 08:22:08.018429 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.017077    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-podinfo\") pod \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") "
Nov 03 08:22:08.018429 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.017098    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-workload-certs\") pod \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\" (UID: \"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4\") "
Nov 03 08:22:08.018429 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.017449    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-workload-socket" pod "a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" (UID: "a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:08.018429 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.017630    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-data" pod "a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" (UID: "a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:08.018429 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.017927    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.018429 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.017950    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.018429 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.017992    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istiod-ca-cert" pod "a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" (UID: "a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:22:08.021338 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.020998    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-workload-certs" pod "a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" (UID: "a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:08.022941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.022915    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-credential-socket" pod "a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" (UID: "a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:08.023048 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.023008    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-podinfo" pod "a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" (UID: "a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:22:08.023945 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.023901    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-kube-api-access-hb9nr" pod "a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" (UID: "a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4"). InnerVolumeSpecName "kube-api-access-hb9nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:08.023945 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.023927    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-token" pod "a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" (UID: "a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:08.025718 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.025591    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-envoy" pod "a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" (UID: "a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:08.036171 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.036144    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="312a9c42-79df-4b2d-b430-0c02d7c37200" path="/var/lib/kubelet/pods/312a9c42-79df-4b2d-b430-0c02d7c37200/volumes"
Nov 03 08:22:08.093306 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:22:08.093269    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d6e143_5bb7_42a8_a5e6_0f14214b7592.slice/crio-conmon-ac0bcc62511738a8bf8144be94bda9ef0d8664b9bea75a800ef8c4b0d9876744.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312a9c42_79df_4b2d_b430_0c02d7c37200.slice/crio-1a0661c239177ea72b63fbd95b1f755bbdaad3e776d38c229d4bba50955fb381\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d6e143_5bb7_42a8_a5e6_0f14214b7592.slice/crio-ac0bcc62511738a8bf8144be94bda9ef0d8664b9bea75a800ef8c4b0d9876744.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 08:22:08.122423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.122395    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.122423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.122425    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb9nr\" (UniqueName: \"kubernetes.io/projected/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-kube-api-access-hb9nr\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.122608 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.122440    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.122608 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.122453    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.122608 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.122467    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.122608 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.122479    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.122608 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.122492    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.248442 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.248404    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"]
Nov 03 08:22:08.289167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.277817    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:22:08.289167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.277948    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:22:08.289167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.278002    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:22:08.303935 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.303560    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf"
Nov 03 08:22:08.354346 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.354307    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4160dd6e-4fae-407d-8573-b2e4d840b9f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4160dd6e-4fae-407d-8573-b2e4d840b9f4-maas-api-xks-ca\") pod \"4160dd6e-4fae-407d-8573-b2e4d840b9f4\" (UID: \"4160dd6e-4fae-407d-8573-b2e4d840b9f4\") "
Nov 03 08:22:08.354493 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.354410    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4160dd6e-4fae-407d-8573-b2e4d840b9f4-kube-api-access-8rfxp\" (UniqueName: \"kubernetes.io/projected/4160dd6e-4fae-407d-8573-b2e4d840b9f4-kube-api-access-8rfxp\") pod \"4160dd6e-4fae-407d-8573-b2e4d840b9f4\" (UID: \"4160dd6e-4fae-407d-8573-b2e4d840b9f4\") "
Nov 03 08:22:08.355011 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.354969    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4160dd6e-4fae-407d-8573-b2e4d840b9f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4160dd6e-4fae-407d-8573-b2e4d840b9f4-maas-api-openshift-ca\") pod \"4160dd6e-4fae-407d-8573-b2e4d840b9f4\" (UID: \"4160dd6e-4fae-407d-8573-b2e4d840b9f4\") "
Nov 03 08:22:08.369820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.355779    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4160dd6e-4fae-407d-8573-b2e4d840b9f4-maas-api-openshift-ca" pod "4160dd6e-4fae-407d-8573-b2e4d840b9f4" (UID: "4160dd6e-4fae-407d-8573-b2e4d840b9f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:22:08.369820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.364462    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4160dd6e-4fae-407d-8573-b2e4d840b9f4-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.386377 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.386222    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4160dd6e-4fae-407d-8573-b2e4d840b9f4-kube-api-access-8rfxp" pod "4160dd6e-4fae-407d-8573-b2e4d840b9f4" (UID: "4160dd6e-4fae-407d-8573-b2e4d840b9f4"). InnerVolumeSpecName "kube-api-access-8rfxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:08.389400 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.389332    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4160dd6e-4fae-407d-8573-b2e4d840b9f4-maas-api-xks-ca" pod "4160dd6e-4fae-407d-8573-b2e4d840b9f4" (UID: "4160dd6e-4fae-407d-8573-b2e4d840b9f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:22:08.482152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.479560    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4160dd6e-4fae-407d-8573-b2e4d840b9f4-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.482152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.479596    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rfxp\" (UniqueName: \"kubernetes.io/projected/4160dd6e-4fae-407d-8573-b2e4d840b9f4-kube-api-access-8rfxp\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.744911 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.744883    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:22:08.866165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.854147    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7" event={"ID":"a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4","Type":"ContainerDied","Data":"85f88274ad26b4db6ba41d1ac50db83853cccc43825e53df2f7edb1cbbb2706a"}
Nov 03 08:22:08.866165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.854223    5070 scope.go:122] "RemoveContainer" containerID="0fa8ebd3a1da0d6692fcdb1427352c97a6148ae8aefc7656bf0c48b4f692ef15"
Nov 03 08:22:08.866165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.854324    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"
Nov 03 08:22:08.903870 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.902216    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn" event={"ID":"8fab335e-1024-4086-92ef-49b496f0abc0","Type":"ContainerStarted","Data":"56b5fd507f150bc77dc9b23955016dd8d479ed5d3d9673f1c4d686954d0c2f30"}
Nov 03 08:22:08.903870 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.902287    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn" event={"ID":"8fab335e-1024-4086-92ef-49b496f0abc0","Type":"ContainerStarted","Data":"93ce3e6108e1b4cae36a596a588460cd5cc1eaf290e8afb729a795970aa90062"}
Nov 03 08:22:08.906902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.905962    5070 generic.go:356] "Generic (PLEG): container finished" podID="56d6e143-5bb7-42a8-a5e6-0f14214b7592" containerID="ac0bcc62511738a8bf8144be94bda9ef0d8664b9bea75a800ef8c4b0d9876744" exitCode=0
Nov 03 08:22:08.906902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.906071    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt" event={"ID":"56d6e143-5bb7-42a8-a5e6-0f14214b7592","Type":"ContainerDied","Data":"ac0bcc62511738a8bf8144be94bda9ef0d8664b9bea75a800ef8c4b0d9876744"}
Nov 03 08:22:08.906902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.906112    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt" event={"ID":"56d6e143-5bb7-42a8-a5e6-0f14214b7592","Type":"ContainerDied","Data":"8220b41807cf5628e84077e51b32779e4b74ece2d1834c777dc6c7124e254707"}
Nov 03 08:22:08.906902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.906186    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"
Nov 03 08:22:08.910381 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.910346    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf"
Nov 03 08:22:08.914556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.913906    5070 scope.go:122] "RemoveContainer" containerID="ac0bcc62511738a8bf8144be94bda9ef0d8664b9bea75a800ef8c4b0d9876744"
Nov 03 08:22:08.914556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.913998    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf" event={"ID":"4160dd6e-4fae-407d-8573-b2e4d840b9f4","Type":"ContainerDied","Data":"926014a817fb1c09fdba169339ececb27f35461a7f48cae51eaba5d4a42021a2"}
Nov 03 08:22:08.914556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.914038    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="926014a817fb1c09fdba169339ececb27f35461a7f48cae51eaba5d4a42021a2"
Nov 03 08:22:08.914556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.914315    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-credential-socket\") pod \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") "
Nov 03 08:22:08.914556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.914375    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-data\") pod \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") "
Nov 03 08:22:08.914556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.914453    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-token\" (UniqueName: \"kubernetes.io/projected/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-token\") pod \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") "
Nov 03 08:22:08.914556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.914503    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-workload-certs\") pod \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") "
Nov 03 08:22:08.916125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.914904    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-data" pod "56d6e143-5bb7-42a8-a5e6-0f14214b7592" (UID: "56d6e143-5bb7-42a8-a5e6-0f14214b7592"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:08.916125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.915310    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d6e143-5bb7-42a8-a5e6-0f14214b7592-kube-api-access-tfxb8\" (UniqueName: \"kubernetes.io/projected/56d6e143-5bb7-42a8-a5e6-0f14214b7592-kube-api-access-tfxb8\") pod \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") "
Nov 03 08:22:08.916125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.915389    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istiod-ca-cert\") pod \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") "
Nov 03 08:22:08.916125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.915451    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-envoy\") pod \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") "
Nov 03 08:22:08.916125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.915588    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-podinfo\") pod \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") "
Nov 03 08:22:08.916125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.915627    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-credential-socket" pod "56d6e143-5bb7-42a8-a5e6-0f14214b7592" (UID: "56d6e143-5bb7-42a8-a5e6-0f14214b7592"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:08.916125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.915872    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-workload-certs" pod "56d6e143-5bb7-42a8-a5e6-0f14214b7592" (UID: "56d6e143-5bb7-42a8-a5e6-0f14214b7592"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:08.916785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.916712    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-workload-socket\") pod \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\" (UID: \"56d6e143-5bb7-42a8-a5e6-0f14214b7592\") "
Nov 03 08:22:08.933992 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.933915    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"]
Nov 03 08:22:08.937993 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.937696    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istiod-ca-cert" pod "56d6e143-5bb7-42a8-a5e6-0f14214b7592" (UID: "56d6e143-5bb7-42a8-a5e6-0f14214b7592"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:22:08.938956 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.938162    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.938956 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.938225    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-envoy" pod "56d6e143-5bb7-42a8-a5e6-0f14214b7592" (UID: "56d6e143-5bb7-42a8-a5e6-0f14214b7592"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:08.938956 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.938248    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.938956 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.938264    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:08.943781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.939725    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-token" pod "56d6e143-5bb7-42a8-a5e6-0f14214b7592" (UID: "56d6e143-5bb7-42a8-a5e6-0f14214b7592"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:08.943781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.939840    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-podinfo" pod "56d6e143-5bb7-42a8-a5e6-0f14214b7592" (UID: "56d6e143-5bb7-42a8-a5e6-0f14214b7592"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:22:08.943781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.941070    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d6e143-5bb7-42a8-a5e6-0f14214b7592-kube-api-access-tfxb8" pod "56d6e143-5bb7-42a8-a5e6-0f14214b7592" (UID: "56d6e143-5bb7-42a8-a5e6-0f14214b7592"). InnerVolumeSpecName "kube-api-access-tfxb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:08.943781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.941311    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-workload-socket" pod "56d6e143-5bb7-42a8-a5e6-0f14214b7592" (UID: "56d6e143-5bb7-42a8-a5e6-0f14214b7592"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:22:08.944776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.944740    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-lqnb7"]
Nov 03 08:22:08.955684 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.955644    5070 scope.go:122] "RemoveContainer" containerID="ac0bcc62511738a8bf8144be94bda9ef0d8664b9bea75a800ef8c4b0d9876744"
Nov 03 08:22:08.972060 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:22:08.972027    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac0bcc62511738a8bf8144be94bda9ef0d8664b9bea75a800ef8c4b0d9876744\": container with ID starting with ac0bcc62511738a8bf8144be94bda9ef0d8664b9bea75a800ef8c4b0d9876744 not found: ID does not exist" containerID="ac0bcc62511738a8bf8144be94bda9ef0d8664b9bea75a800ef8c4b0d9876744"
Nov 03 08:22:08.972296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:08.972252    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac0bcc62511738a8bf8144be94bda9ef0d8664b9bea75a800ef8c4b0d9876744"} err="failed to get container status \"ac0bcc62511738a8bf8144be94bda9ef0d8664b9bea75a800ef8c4b0d9876744\": rpc error: code = NotFound desc = could not find container \"ac0bcc62511738a8bf8144be94bda9ef0d8664b9bea75a800ef8c4b0d9876744\": container with ID starting with ac0bcc62511738a8bf8144be94bda9ef0d8664b9bea75a800ef8c4b0d9876744 not found: ID does not exist"
Nov 03 08:22:09.053853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.053485    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:09.053853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.053522    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:09.053853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.053537    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfxb8\" (UniqueName: \"kubernetes.io/projected/56d6e143-5bb7-42a8-a5e6-0f14214b7592-kube-api-access-tfxb8\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:09.053853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.053551    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:09.053853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.053565    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:09.053853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.053579    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d6e143-5bb7-42a8-a5e6-0f14214b7592-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:09.087578 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.087515    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn" podStartSLOduration=245.087492839 podStartE2EDuration="4m5.087492839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:22:08.984005642 +0000 UTC m=+1727.098469574" watchObservedRunningTime="2026-11-03 08:22:09.087492839 +0000 UTC m=+1727.201956771"
Nov 03 08:22:09.446492 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.446453    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"]
Nov 03 08:22:09.454374 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.454344    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-j6mlt"]
Nov 03 08:22:09.722853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.705836    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:09.722853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.706470    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Nov 03 08:22:09.722853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.706619    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn" podUID="8fab335e-1024-4086-92ef-49b496f0abc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Nov 03 08:22:09.752803 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.752752    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq"
Nov 03 08:22:09.853577 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.853546    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6c0b1f1-402f-4eb0-adae-9aa037a900e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6c0b1f1-402f-4eb0-adae-9aa037a900e5-maas-api-openshift-ca\") pod \"f6c0b1f1-402f-4eb0-adae-9aa037a900e5\" (UID: \"f6c0b1f1-402f-4eb0-adae-9aa037a900e5\") "
Nov 03 08:22:09.853792 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.853674    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6c0b1f1-402f-4eb0-adae-9aa037a900e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6c0b1f1-402f-4eb0-adae-9aa037a900e5-maas-api-xks-ca\") pod \"f6c0b1f1-402f-4eb0-adae-9aa037a900e5\" (UID: \"f6c0b1f1-402f-4eb0-adae-9aa037a900e5\") "
Nov 03 08:22:09.853792 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.853725    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c0b1f1-402f-4eb0-adae-9aa037a900e5-kube-api-access-vzrct\" (UniqueName: \"kubernetes.io/projected/f6c0b1f1-402f-4eb0-adae-9aa037a900e5-kube-api-access-vzrct\") pod \"f6c0b1f1-402f-4eb0-adae-9aa037a900e5\" (UID: \"f6c0b1f1-402f-4eb0-adae-9aa037a900e5\") "
Nov 03 08:22:09.853994 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.853961    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c0b1f1-402f-4eb0-adae-9aa037a900e5-maas-api-openshift-ca" pod "f6c0b1f1-402f-4eb0-adae-9aa037a900e5" (UID: "f6c0b1f1-402f-4eb0-adae-9aa037a900e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:22:09.854578 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.854553    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6c0b1f1-402f-4eb0-adae-9aa037a900e5-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:09.880779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.873985    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c0b1f1-402f-4eb0-adae-9aa037a900e5-kube-api-access-vzrct" pod "f6c0b1f1-402f-4eb0-adae-9aa037a900e5" (UID: "f6c0b1f1-402f-4eb0-adae-9aa037a900e5"). InnerVolumeSpecName "kube-api-access-vzrct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:09.909787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.909279    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c0b1f1-402f-4eb0-adae-9aa037a900e5-maas-api-xks-ca" pod "f6c0b1f1-402f-4eb0-adae-9aa037a900e5" (UID: "f6c0b1f1-402f-4eb0-adae-9aa037a900e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:22:09.922849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.922751    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq"
Nov 03 08:22:09.922849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.922813    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq" event={"ID":"f6c0b1f1-402f-4eb0-adae-9aa037a900e5","Type":"ContainerDied","Data":"345878d74e9b26319dd185193e1c7795209ad5e030fc6d1702619b3f3f98b8c9"}
Nov 03 08:22:09.923068 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.922869    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="345878d74e9b26319dd185193e1c7795209ad5e030fc6d1702619b3f3f98b8c9"
Nov 03 08:22:09.979781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.975941    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6c0b1f1-402f-4eb0-adae-9aa037a900e5-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:09.979781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:09.975982    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzrct\" (UniqueName: \"kubernetes.io/projected/f6c0b1f1-402f-4eb0-adae-9aa037a900e5-kube-api-access-vzrct\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:10.029394 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:10.029360    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56d6e143-5bb7-42a8-a5e6-0f14214b7592" path="/var/lib/kubelet/pods/56d6e143-5bb7-42a8-a5e6-0f14214b7592/volumes"
Nov 03 08:22:10.030047 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:10.029917    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" path="/var/lib/kubelet/pods/a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4/volumes"
Nov 03 08:22:10.713790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:10.709117    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Nov 03 08:22:10.713790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:10.709177    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn" podUID="8fab335e-1024-4086-92ef-49b496f0abc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Nov 03 08:22:11.707191 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:11.707145    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Nov 03 08:22:11.707821 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:11.707786    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn" podUID="8fab335e-1024-4086-92ef-49b496f0abc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Nov 03 08:22:12.704502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:12.704422    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Nov 03 08:22:12.704502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:12.704472    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn" podUID="8fab335e-1024-4086-92ef-49b496f0abc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Nov 03 08:22:13.710778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:13.708886    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Nov 03 08:22:13.710778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:13.708965    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn" podUID="8fab335e-1024-4086-92ef-49b496f0abc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Nov 03 08:22:13.995266 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:13.995191    5070 scope.go:122] "RemoveContainer" containerID="82c16f70c7cb07913f06ba57165da72e1281fe64013d19b173097cafa3e89784"
Nov 03 08:22:14.718716 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:14.711942    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Nov 03 08:22:14.718716 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:14.712017    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn" podUID="8fab335e-1024-4086-92ef-49b496f0abc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Nov 03 08:22:15.711782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:15.705409    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body=
Nov 03 08:22:15.711782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:15.705482    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn" podUID="8fab335e-1024-4086-92ef-49b496f0abc0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused"
Nov 03 08:22:16.881640 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:16.881602    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:16.882321 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:16.882111    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:16.883000 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:16.882977    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:22:17.429478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:17.420128    5070 patch_prober.go:29] interesting pod/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:22:17.429478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:17.420211    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b" podUID="8e96f256-aa30-4c3e-9d97-d829c96d13ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:22:19.176790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.167536    5070 generic.go:356] "Generic (PLEG): container finished" podID="8e96f256-aa30-4c3e-9d97-d829c96d13ac" containerID="f05d3136eb9be54b7c1b46679e84dab6c1c0ede5c0327ab9abf0ed279871202e" exitCode=0
Nov 03 08:22:19.176790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.167678    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b" event={"ID":"8e96f256-aa30-4c3e-9d97-d829c96d13ac","Type":"ContainerDied","Data":"f05d3136eb9be54b7c1b46679e84dab6c1c0ede5c0327ab9abf0ed279871202e"}
Nov 03 08:22:19.176790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.168292    5070 scope.go:122] "RemoveContainer" containerID="f05d3136eb9be54b7c1b46679e84dab6c1c0ede5c0327ab9abf0ed279871202e"
Nov 03 08:22:19.182239 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.182196    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:22:19.182498 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.182472    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:22:19.183155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.182655    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:22:19.385584 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.385556    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g"]
Nov 03 08:22:19.386434 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.386416    5070 cpu_manager.go:414] "containerMap: removing container" podUID="4160dd6e-4fae-407d-8573-b2e4d840b9f4" containerName="revoke-keys"
Nov 03 08:22:19.386490 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.386437    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4160dd6e-4fae-407d-8573-b2e4d840b9f4" containerName="revoke-keys"
Nov 03 08:22:19.386490 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.386455    5070 cpu_manager.go:414] "containerMap: removing container" podUID="f6c0b1f1-402f-4eb0-adae-9aa037a900e5" containerName="revoke-keys"
Nov 03 08:22:19.386490 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.386460    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c0b1f1-402f-4eb0-adae-9aa037a900e5" containerName="revoke-keys"
Nov 03 08:22:19.386490 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.386469    5070 cpu_manager.go:414] "containerMap: removing container" podUID="a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" containerName="istio-proxy"
Nov 03 08:22:19.386490 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.386474    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" containerName="istio-proxy"
Nov 03 08:22:19.387439 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.386495    5070 cpu_manager.go:414] "containerMap: removing container" podUID="56d6e143-5bb7-42a8-a5e6-0f14214b7592" containerName="istio-proxy"
Nov 03 08:22:19.387439 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.386500    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d6e143-5bb7-42a8-a5e6-0f14214b7592" containerName="istio-proxy"
Nov 03 08:22:19.387439 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.386518    5070 cpu_manager.go:414] "containerMap: removing container" podUID="312a9c42-79df-4b2d-b430-0c02d7c37200" containerName="istio-proxy"
Nov 03 08:22:19.387439 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.386523    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="312a9c42-79df-4b2d-b430-0c02d7c37200" containerName="istio-proxy"
Nov 03 08:22:19.387439 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.386635    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c0b1f1-402f-4eb0-adae-9aa037a900e5" containerName="revoke-keys"
Nov 03 08:22:19.387439 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.386645    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e6709c-fa81-4a1c-81ee-e6cbecd5c7c4" containerName="istio-proxy"
Nov 03 08:22:19.387439 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.386653    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d6e143-5bb7-42a8-a5e6-0f14214b7592" containerName="istio-proxy"
Nov 03 08:22:19.387439 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.386668    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="4160dd6e-4fae-407d-8573-b2e4d840b9f4" containerName="revoke-keys"
Nov 03 08:22:19.389198 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.389179    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g"
Nov 03 08:22:19.393088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.392515    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:22:19.393481 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.393364    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:22:19.393582 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.393546    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:22:19.393648 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.393593    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:22:19.400521 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.400498    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g"]
Nov 03 08:22:19.509493 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.509401    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g\" (UID: \"d08f3a4a-cb5b-4a9b-baf8-674ffecf7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g"
Nov 03 08:22:19.509714 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.509515    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g\" (UID: \"d08f3a4a-cb5b-4a9b-baf8-674ffecf7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g"
Nov 03 08:22:19.509714 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.509598    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xck59\" (UniqueName: \"kubernetes.io/projected/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287-kube-api-access-xck59\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g\" (UID: \"d08f3a4a-cb5b-4a9b-baf8-674ffecf7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g"
Nov 03 08:22:19.612778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.612730    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g\" (UID: \"d08f3a4a-cb5b-4a9b-baf8-674ffecf7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g"
Nov 03 08:22:19.613221 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.613065    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g\" (UID: \"d08f3a4a-cb5b-4a9b-baf8-674ffecf7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g"
Nov 03 08:22:19.613221 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.613142    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xck59\" (UniqueName: \"kubernetes.io/projected/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287-kube-api-access-xck59\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g\" (UID: \"d08f3a4a-cb5b-4a9b-baf8-674ffecf7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g"
Nov 03 08:22:19.614804 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.614738    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g\" (UID: \"d08f3a4a-cb5b-4a9b-baf8-674ffecf7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g"
Nov 03 08:22:19.622926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.621624    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g\" (UID: \"d08f3a4a-cb5b-4a9b-baf8-674ffecf7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g"
Nov 03 08:22:19.626418 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.626393    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xck59\" (UniqueName: \"kubernetes.io/projected/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287-kube-api-access-xck59\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g\" (UID: \"d08f3a4a-cb5b-4a9b-baf8-674ffecf7287\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g"
Nov 03 08:22:19.725800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:19.723190    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g"
Nov 03 08:22:20.191853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:20.191820    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b" event={"ID":"8e96f256-aa30-4c3e-9d97-d829c96d13ac","Type":"ContainerStarted","Data":"3a6ebaa0a02caaeb1687c0a06c85877a161b816fa8de658d0e44407a72b8d44b"}
Nov 03 08:22:20.209803 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:22:20.207710    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08f3a4a_cb5b_4a9b_baf8_674ffecf7287.slice/crio-38878d6fe42b89c39b7cb227086e8d79923af00c205d97428a71fae9701c7e95 WatchSource:0}: Error finding container 38878d6fe42b89c39b7cb227086e8d79923af00c205d97428a71fae9701c7e95: Status 404 returned error can't find the container with id 38878d6fe42b89c39b7cb227086e8d79923af00c205d97428a71fae9701c7e95
Nov 03 08:22:20.209803 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:20.208959    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g"]
Nov 03 08:22:20.652088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:20.641000    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:22:20.652088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:20.643025    5070 patch_prober.go:29] interesting pod/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 03 08:22:20.652088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:20.643067    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b" podUID="8e96f256-aa30-4c3e-9d97-d829c96d13ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 03 08:22:21.202849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:21.202811    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g" event={"ID":"d08f3a4a-cb5b-4a9b-baf8-674ffecf7287","Type":"ContainerStarted","Data":"83f57ad4dcc4607eea144ad3afb0ce83ad199c5b3db06de70697a3eb54913554"}
Nov 03 08:22:21.203272 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:21.202877    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g" event={"ID":"d08f3a4a-cb5b-4a9b-baf8-674ffecf7287","Type":"ContainerStarted","Data":"38878d6fe42b89c39b7cb227086e8d79923af00c205d97428a71fae9701c7e95"}
Nov 03 08:22:21.216915 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:21.216862    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g" podStartSLOduration=2.216845445 podStartE2EDuration="2.216845445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:22:21.215496508 +0000 UTC m=+1739.329960439" watchObservedRunningTime="2026-11-03 08:22:21.216845445 +0000 UTC m=+1739.331309377"
Nov 03 08:22:21.641536 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:21.641497    5070 patch_prober.go:29] interesting pod/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 03 08:22:21.641709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:21.641558    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b" podUID="8e96f256-aa30-4c3e-9d97-d829c96d13ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 03 08:22:22.052018 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:22:22.051940    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:22:22.214155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:22.214116    5070 generic.go:356] "Generic (PLEG): container finished" podID="d08f3a4a-cb5b-4a9b-baf8-674ffecf7287" containerID="83f57ad4dcc4607eea144ad3afb0ce83ad199c5b3db06de70697a3eb54913554" exitCode=0
Nov 03 08:22:22.214543 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:22.214200    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g" event={"ID":"d08f3a4a-cb5b-4a9b-baf8-674ffecf7287","Type":"ContainerDied","Data":"83f57ad4dcc4607eea144ad3afb0ce83ad199c5b3db06de70697a3eb54913554"}
Nov 03 08:22:22.640701 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:22.640663    5070 patch_prober.go:29] interesting pod/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 03 08:22:22.640892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:22.640724    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b" podUID="8e96f256-aa30-4c3e-9d97-d829c96d13ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 03 08:22:23.632055 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:23.632023    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g"
Nov 03 08:22:23.641634 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:23.641600    5070 patch_prober.go:29] interesting pod/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 03 08:22:23.641841 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:23.641672    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b" podUID="8e96f256-aa30-4c3e-9d97-d829c96d13ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 03 08:22:23.777834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:23.760527    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287-maas-api-openshift-ca\") pod \"d08f3a4a-cb5b-4a9b-baf8-674ffecf7287\" (UID: \"d08f3a4a-cb5b-4a9b-baf8-674ffecf7287\") "
Nov 03 08:22:23.777834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:23.760747    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287-maas-api-xks-ca\") pod \"d08f3a4a-cb5b-4a9b-baf8-674ffecf7287\" (UID: \"d08f3a4a-cb5b-4a9b-baf8-674ffecf7287\") "
Nov 03 08:22:23.777834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:23.760816    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287-kube-api-access-xck59\" (UniqueName: \"kubernetes.io/projected/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287-kube-api-access-xck59\") pod \"d08f3a4a-cb5b-4a9b-baf8-674ffecf7287\" (UID: \"d08f3a4a-cb5b-4a9b-baf8-674ffecf7287\") "
Nov 03 08:22:23.783245 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:23.783210    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287-maas-api-openshift-ca" pod "d08f3a4a-cb5b-4a9b-baf8-674ffecf7287" (UID: "d08f3a4a-cb5b-4a9b-baf8-674ffecf7287"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:22:23.788942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:23.787929    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287-kube-api-access-xck59" pod "d08f3a4a-cb5b-4a9b-baf8-674ffecf7287" (UID: "d08f3a4a-cb5b-4a9b-baf8-674ffecf7287"). InnerVolumeSpecName "kube-api-access-xck59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:23.788942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:23.788422    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287-maas-api-xks-ca" pod "d08f3a4a-cb5b-4a9b-baf8-674ffecf7287" (UID: "d08f3a4a-cb5b-4a9b-baf8-674ffecf7287"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:22:23.882420 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:23.882387    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:23.882420 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:23.882421    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xck59\" (UniqueName: \"kubernetes.io/projected/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287-kube-api-access-xck59\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:23.882630 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:23.882436    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:24.253373 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:24.253339    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g"
Nov 03 08:22:24.253578 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:24.253371    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g" event={"ID":"d08f3a4a-cb5b-4a9b-baf8-674ffecf7287","Type":"ContainerDied","Data":"38878d6fe42b89c39b7cb227086e8d79923af00c205d97428a71fae9701c7e95"}
Nov 03 08:22:24.253578 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:24.253423    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38878d6fe42b89c39b7cb227086e8d79923af00c205d97428a71fae9701c7e95"
Nov 03 08:22:24.652652 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:24.652608    5070 patch_prober.go:29] interesting pod/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 03 08:22:24.653209 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:24.652676    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b" podUID="8e96f256-aa30-4c3e-9d97-d829c96d13ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 03 08:22:25.641825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:25.641788    5070 patch_prober.go:29] interesting pod/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body=
Nov 03 08:22:25.642025 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:25.641841    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b" podUID="8e96f256-aa30-4c3e-9d97-d829c96d13ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused"
Nov 03 08:22:26.645236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:26.645206    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:22:26.645769 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:26.645735    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:22:26.646533 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:26.646514    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:22:27.819417 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:27.819383    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf"]
Nov 03 08:22:27.824577 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:27.824537    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-7a6481-6mbpf"]
Nov 03 08:22:28.095067 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:28.094983    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4160dd6e-4fae-407d-8573-b2e4d840b9f4" path="/var/lib/kubelet/pods/4160dd6e-4fae-407d-8573-b2e4d840b9f4/volumes"
Nov 03 08:22:31.001859 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:31.001816    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq"]
Nov 03 08:22:31.027783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:31.027675    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ba69a0-kkjrq"]
Nov 03 08:22:32.029856 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:32.029822    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6c0b1f1-402f-4eb0-adae-9aa037a900e5" path="/var/lib/kubelet/pods/f6c0b1f1-402f-4eb0-adae-9aa037a900e5/volumes"
Nov 03 08:22:35.296826 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:35.295561    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw"]
Nov 03 08:22:35.298742 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:35.298713    5070 cpu_manager.go:414] "containerMap: removing container" podUID="d08f3a4a-cb5b-4a9b-baf8-674ffecf7287" containerName="revoke-keys"
Nov 03 08:22:35.298939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:35.298923    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d08f3a4a-cb5b-4a9b-baf8-674ffecf7287" containerName="revoke-keys"
Nov 03 08:22:35.299261 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:35.299242    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="d08f3a4a-cb5b-4a9b-baf8-674ffecf7287" containerName="revoke-keys"
Nov 03 08:22:35.353924 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:35.353892    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw"]
Nov 03 08:22:35.354289 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:35.354267    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw"
Nov 03 08:22:35.382805 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:35.376201    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:22:35.382805 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:35.381054    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:22:35.382805 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:35.381128    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:22:35.407780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:35.401078    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw\" (UID: \"f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw"
Nov 03 08:22:35.407780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:35.401488    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xjwm\" (UniqueName: \"kubernetes.io/projected/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6-kube-api-access-9xjwm\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw\" (UID: \"f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw"
Nov 03 08:22:35.407780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:35.401551    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw\" (UID: \"f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw"
Nov 03 08:22:35.483194 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:35.474734    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:22:35.503798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:35.503744    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xjwm\" (UniqueName: \"kubernetes.io/projected/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6-kube-api-access-9xjwm\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw\" (UID: \"f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw"
Nov 03 08:22:35.504062 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:35.504040    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw\" (UID: \"f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw"
Nov 03 08:22:35.504346 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:35.504323    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw\" (UID: \"f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw"
Nov 03 08:22:35.505486 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:35.505464    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw\" (UID: \"f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw"
Nov 03 08:22:35.512419 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:35.512339    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw\" (UID: \"f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw"
Nov 03 08:22:35.518915 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:35.518897    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xjwm\" (UniqueName: \"kubernetes.io/projected/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6-kube-api-access-9xjwm\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw\" (UID: \"f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw"
Nov 03 08:22:35.800555 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:35.800526    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw"
Nov 03 08:22:36.102633 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:36.102600    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw"]
Nov 03 08:22:36.103109 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:22:36.103085    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf53bc5e7_aae7_46f4_b95f_8ed3c60ec4d6.slice/crio-b5d395dfbc8e4f1bdd83f775ec80e69188f420e35e98541bcb5f9d1c440c8971 WatchSource:0}: Error finding container b5d395dfbc8e4f1bdd83f775ec80e69188f420e35e98541bcb5f9d1c440c8971: Status 404 returned error can't find the container with id b5d395dfbc8e4f1bdd83f775ec80e69188f420e35e98541bcb5f9d1c440c8971
Nov 03 08:22:36.706242 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:36.706204    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw" event={"ID":"f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6","Type":"ContainerStarted","Data":"14ea11090aced2a25c739bcb463a9014561e8b797d8eba04ec2d558d6d30119f"}
Nov 03 08:22:36.706647 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:36.706263    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw" event={"ID":"f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6","Type":"ContainerStarted","Data":"b5d395dfbc8e4f1bdd83f775ec80e69188f420e35e98541bcb5f9d1c440c8971"}
Nov 03 08:22:36.722681 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:36.722625    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw" podStartSLOduration=1.722609019 podStartE2EDuration="1.722609019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:22:36.720456368 +0000 UTC m=+1754.834920299" watchObservedRunningTime="2026-11-03 08:22:36.722609019 +0000 UTC m=+1754.837072953"
Nov 03 08:22:37.799881 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:37.799796    5070 generic.go:356] "Generic (PLEG): container finished" podID="f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6" containerID="14ea11090aced2a25c739bcb463a9014561e8b797d8eba04ec2d558d6d30119f" exitCode=0
Nov 03 08:22:37.800853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:37.800618    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw" event={"ID":"f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6","Type":"ContainerDied","Data":"14ea11090aced2a25c739bcb463a9014561e8b797d8eba04ec2d558d6d30119f"}
Nov 03 08:22:39.601050 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.601019    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd"]
Nov 03 08:22:39.615540 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.615517    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd"
Nov 03 08:22:39.617202 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.617174    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd"]
Nov 03 08:22:39.629899 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.629169    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw"
Nov 03 08:22:39.700074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.697873    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6-kube-api-access-9xjwm\" (UniqueName: \"kubernetes.io/projected/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6-kube-api-access-9xjwm\") pod \"f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6\" (UID: \"f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6\") "
Nov 03 08:22:39.700074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.697967    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6-maas-api-openshift-ca\") pod \"f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6\" (UID: \"f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6\") "
Nov 03 08:22:39.700074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.698317    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6-maas-api-xks-ca\") pod \"f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6\" (UID: \"f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6\") "
Nov 03 08:22:39.700074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.698706    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd\" (UID: \"3b44fb15-f8cb-4bd2-91f8-9d784d1a0083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd"
Nov 03 08:22:39.703078 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.701395    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6-maas-api-openshift-ca" pod "f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6" (UID: "f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:22:39.704100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.703882    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzhtd\" (UniqueName: \"kubernetes.io/projected/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083-kube-api-access-tzhtd\") pod \"maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd\" (UID: \"3b44fb15-f8cb-4bd2-91f8-9d784d1a0083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd"
Nov 03 08:22:39.704100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.704062    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd\" (UID: \"3b44fb15-f8cb-4bd2-91f8-9d784d1a0083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd"
Nov 03 08:22:39.704385 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.704355    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:39.705416 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.705371    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6-kube-api-access-9xjwm" pod "f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6" (UID: "f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6"). InnerVolumeSpecName "kube-api-access-9xjwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:39.713357 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.713254    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6-maas-api-xks-ca" pod "f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6" (UID: "f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:22:39.806263 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.806230    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzhtd\" (UniqueName: \"kubernetes.io/projected/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083-kube-api-access-tzhtd\") pod \"maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd\" (UID: \"3b44fb15-f8cb-4bd2-91f8-9d784d1a0083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd"
Nov 03 08:22:39.806461 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.806335    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd\" (UID: \"3b44fb15-f8cb-4bd2-91f8-9d784d1a0083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd"
Nov 03 08:22:39.806461 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.806452    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd\" (UID: \"3b44fb15-f8cb-4bd2-91f8-9d784d1a0083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd"
Nov 03 08:22:39.806604 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.806569    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xjwm\" (UniqueName: \"kubernetes.io/projected/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6-kube-api-access-9xjwm\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:39.806604 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.806581    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:39.807462 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.807432    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd\" (UID: \"3b44fb15-f8cb-4bd2-91f8-9d784d1a0083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd"
Nov 03 08:22:39.810135 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.810069    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd\" (UID: \"3b44fb15-f8cb-4bd2-91f8-9d784d1a0083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd"
Nov 03 08:22:39.819043 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.819020    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzhtd\" (UniqueName: \"kubernetes.io/projected/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083-kube-api-access-tzhtd\") pod \"maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd\" (UID: \"3b44fb15-f8cb-4bd2-91f8-9d784d1a0083\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd"
Nov 03 08:22:39.827035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.827013    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw"
Nov 03 08:22:39.827144 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.827036    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw" event={"ID":"f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6","Type":"ContainerDied","Data":"b5d395dfbc8e4f1bdd83f775ec80e69188f420e35e98541bcb5f9d1c440c8971"}
Nov 03 08:22:39.827144 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.827091    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5d395dfbc8e4f1bdd83f775ec80e69188f420e35e98541bcb5f9d1c440c8971"
Nov 03 08:22:39.990428 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:39.990391    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd"
Nov 03 08:22:40.290375 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:40.290344    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd"]
Nov 03 08:22:40.295092 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:22:40.295064    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b44fb15_f8cb_4bd2_91f8_9d784d1a0083.slice/crio-2d369da6436a13a939eaa65ad90b3895e2226d39ca9e815991d72bf2a3233037 WatchSource:0}: Error finding container 2d369da6436a13a939eaa65ad90b3895e2226d39ca9e815991d72bf2a3233037: Status 404 returned error can't find the container with id 2d369da6436a13a939eaa65ad90b3895e2226d39ca9e815991d72bf2a3233037
Nov 03 08:22:40.853411 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:40.852907    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd" event={"ID":"3b44fb15-f8cb-4bd2-91f8-9d784d1a0083","Type":"ContainerStarted","Data":"2d369da6436a13a939eaa65ad90b3895e2226d39ca9e815991d72bf2a3233037"}
Nov 03 08:22:41.864798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:41.864721    5070 generic.go:356] "Generic (PLEG): container finished" podID="3b44fb15-f8cb-4bd2-91f8-9d784d1a0083" containerID="c0b057e62081994633a8b23d17c728488140db06194115ecb2facc7b4fbfd23f" exitCode=0
Nov 03 08:22:41.864798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:41.864774    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd" event={"ID":"3b44fb15-f8cb-4bd2-91f8-9d784d1a0083","Type":"ContainerDied","Data":"c0b057e62081994633a8b23d17c728488140db06194115ecb2facc7b4fbfd23f"}
Nov 03 08:22:43.247974 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:43.243957    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:22:43.247974 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:43.244321    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t" podUID="66d4513c-da3f-430c-91e3-bb8d782d5917" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded"
Nov 03 08:22:43.355957 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:43.354907    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:22:43.385959 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:43.385754    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:22:43.397230 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:43.387379    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:22:44.257115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:44.257068    5070 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/health\": context deadline exceeded"
Nov 03 08:22:44.396885 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:44.396837    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:22:44.517941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:44.517835    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:22:44.544594 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:44.544558    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd"
Nov 03 08:22:44.844505 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:44.820654    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083-maas-api-openshift-ca\") pod \"3b44fb15-f8cb-4bd2-91f8-9d784d1a0083\" (UID: \"3b44fb15-f8cb-4bd2-91f8-9d784d1a0083\") "
Nov 03 08:22:44.844505 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:44.820888    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083-kube-api-access-tzhtd\" (UniqueName: \"kubernetes.io/projected/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083-kube-api-access-tzhtd\") pod \"3b44fb15-f8cb-4bd2-91f8-9d784d1a0083\" (UID: \"3b44fb15-f8cb-4bd2-91f8-9d784d1a0083\") "
Nov 03 08:22:44.844505 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:44.820976    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083-maas-api-xks-ca\") pod \"3b44fb15-f8cb-4bd2-91f8-9d784d1a0083\" (UID: \"3b44fb15-f8cb-4bd2-91f8-9d784d1a0083\") "
Nov 03 08:22:44.906703 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:44.906663    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083-maas-api-openshift-ca" pod "3b44fb15-f8cb-4bd2-91f8-9d784d1a0083" (UID: "3b44fb15-f8cb-4bd2-91f8-9d784d1a0083"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:22:45.063039 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:45.063000    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:45.064076 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:45.064027    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083-kube-api-access-tzhtd" pod "3b44fb15-f8cb-4bd2-91f8-9d784d1a0083" (UID: "3b44fb15-f8cb-4bd2-91f8-9d784d1a0083"). InnerVolumeSpecName "kube-api-access-tzhtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:22:45.073271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:45.073232    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083-maas-api-xks-ca" pod "3b44fb15-f8cb-4bd2-91f8-9d784d1a0083" (UID: "3b44fb15-f8cb-4bd2-91f8-9d784d1a0083"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:22:45.153828 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:22:45.153794    5070 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.123s"
Nov 03 08:22:45.184269 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:45.184241    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:22:45.186807 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:45.181626    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzhtd\" (UniqueName: \"kubernetes.io/projected/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083-kube-api-access-tzhtd\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:45.187148 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:45.187127    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:22:45.236305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:45.236179    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:22:45.397929 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:45.397885    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:22:45.447591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:45.447504    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:22:45.480492 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:45.480456    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd"
Nov 03 08:22:45.481914 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:45.481257    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd" event={"ID":"3b44fb15-f8cb-4bd2-91f8-9d784d1a0083","Type":"ContainerDied","Data":"2d369da6436a13a939eaa65ad90b3895e2226d39ca9e815991d72bf2a3233037"}
Nov 03 08:22:45.481914 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:45.481304    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d369da6436a13a939eaa65ad90b3895e2226d39ca9e815991d72bf2a3233037"
Nov 03 08:22:48.571720 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:48.571633    5070 generic.go:356] "Generic (PLEG): container finished" podID="08fbd019-0105-4baf-8c74-ae606e368876" containerID="ed709d0dfc6afb094a5c7e3f852e8991cd96e0de1ae4f9ba5f1eca9f0d764ba9" exitCode=0
Nov 03 08:22:48.571720 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:48.571711    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx" event={"ID":"08fbd019-0105-4baf-8c74-ae606e368876","Type":"ContainerDied","Data":"ed709d0dfc6afb094a5c7e3f852e8991cd96e0de1ae4f9ba5f1eca9f0d764ba9"}
Nov 03 08:22:48.572967 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:48.572263    5070 scope.go:122] "RemoveContainer" containerID="ed709d0dfc6afb094a5c7e3f852e8991cd96e0de1ae4f9ba5f1eca9f0d764ba9"
Nov 03 08:22:48.574325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:48.574295    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:22:48.574389 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:48.574370    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:22:48.574427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:48.574405    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:22:49.591649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:49.591611    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx" event={"ID":"08fbd019-0105-4baf-8c74-ae606e368876","Type":"ContainerStarted","Data":"05b4a6bb2fe35070a26bc0a59f20826af5b461072cdab1dea6b526d77dcfa536"}
Nov 03 08:22:50.128806 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:50.126851    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:22:50.580902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:50.580857    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:22:50.581258 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:50.581227    5070 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 03 08:22:50.581347 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:50.581306    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx" podUID="08fbd019-0105-4baf-8c74-ae606e368876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 03 08:22:51.581477 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:51.581219    5070 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 03 08:22:51.581477 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:51.581294    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx" podUID="08fbd019-0105-4baf-8c74-ae606e368876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 03 08:22:52.143812 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:22:52.143769    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:22:52.581188 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:52.581146    5070 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 03 08:22:52.581374 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:52.581218    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx" podUID="08fbd019-0105-4baf-8c74-ae606e368876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 03 08:22:53.600787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:53.580154    5070 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 03 08:22:53.600787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:53.580217    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx" podUID="08fbd019-0105-4baf-8c74-ae606e368876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 03 08:22:53.882502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:53.881841    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g"]
Nov 03 08:22:53.898397 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:53.898341    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a60004-wr89g"]
Nov 03 08:22:54.153827 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:54.144687    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d08f3a4a-cb5b-4a9b-baf8-674ffecf7287" path="/var/lib/kubelet/pods/d08f3a4a-cb5b-4a9b-baf8-674ffecf7287/volumes"
Nov 03 08:22:54.582222 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:54.582136    5070 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 03 08:22:54.582496 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:54.582465    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx" podUID="08fbd019-0105-4baf-8c74-ae606e368876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 03 08:22:55.579355 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:55.579313    5070 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused" start-of-body=
Nov 03 08:22:55.579791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:55.579395    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx" podUID="08fbd019-0105-4baf-8c74-ae606e368876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.155:15021/healthz/ready\": dial tcp 10.132.0.155:15021: connect: connection refused"
Nov 03 08:22:56.627169 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:56.627126    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:22:56.630899 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:22:56.630870    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:23:00.920911 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:00.920878    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"]
Nov 03 08:23:00.921701 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:00.921658    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx" podUID="08fbd019-0105-4baf-8c74-ae606e368876" containerName="istio-proxy" containerID="cri-o://05b4a6bb2fe35070a26bc0a59f20826af5b461072cdab1dea6b526d77dcfa536" gracePeriod=30
Nov 03 08:23:01.455181 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.455149    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt"]
Nov 03 08:23:01.456167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.456141    5070 cpu_manager.go:414] "containerMap: removing container" podUID="3b44fb15-f8cb-4bd2-91f8-9d784d1a0083" containerName="revoke-keys"
Nov 03 08:23:01.456167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.456166    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b44fb15-f8cb-4bd2-91f8-9d784d1a0083" containerName="revoke-keys"
Nov 03 08:23:01.456391 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.456202    5070 cpu_manager.go:414] "containerMap: removing container" podUID="f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6" containerName="revoke-keys"
Nov 03 08:23:01.456391 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.456211    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6" containerName="revoke-keys"
Nov 03 08:23:01.456391 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.456380    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b44fb15-f8cb-4bd2-91f8-9d784d1a0083" containerName="revoke-keys"
Nov 03 08:23:01.456561 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.456395    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6" containerName="revoke-keys"
Nov 03 08:23:01.460863 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.460840    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt"
Nov 03 08:23:01.469626 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.468972    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:23:01.529792 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.512886    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt"]
Nov 03 08:23:01.545782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.545268    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:23:01.545782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.545409    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:23:01.569178 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.568820    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:23:01.614239 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.614204    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e569cd-e7c8-4a6c-a385-5a352a7e6b23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt\" (UID: \"38e569cd-e7c8-4a6c-a385-5a352a7e6b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt"
Nov 03 08:23:01.614535 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.614513    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7zt9\" (UniqueName: \"kubernetes.io/projected/38e569cd-e7c8-4a6c-a385-5a352a7e6b23-kube-api-access-s7zt9\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt\" (UID: \"38e569cd-e7c8-4a6c-a385-5a352a7e6b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt"
Nov 03 08:23:01.614635 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.614616    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e569cd-e7c8-4a6c-a385-5a352a7e6b23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt\" (UID: \"38e569cd-e7c8-4a6c-a385-5a352a7e6b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt"
Nov 03 08:23:01.715667 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.715592    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7zt9\" (UniqueName: \"kubernetes.io/projected/38e569cd-e7c8-4a6c-a385-5a352a7e6b23-kube-api-access-s7zt9\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt\" (UID: \"38e569cd-e7c8-4a6c-a385-5a352a7e6b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt"
Nov 03 08:23:01.715853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.715670    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e569cd-e7c8-4a6c-a385-5a352a7e6b23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt\" (UID: \"38e569cd-e7c8-4a6c-a385-5a352a7e6b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt"
Nov 03 08:23:01.715853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.715806    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e569cd-e7c8-4a6c-a385-5a352a7e6b23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt\" (UID: \"38e569cd-e7c8-4a6c-a385-5a352a7e6b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt"
Nov 03 08:23:01.716334 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.716308    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e569cd-e7c8-4a6c-a385-5a352a7e6b23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt\" (UID: \"38e569cd-e7c8-4a6c-a385-5a352a7e6b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt"
Nov 03 08:23:01.718361 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.718338    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e569cd-e7c8-4a6c-a385-5a352a7e6b23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt\" (UID: \"38e569cd-e7c8-4a6c-a385-5a352a7e6b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt"
Nov 03 08:23:01.722474 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.722442    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7zt9\" (UniqueName: \"kubernetes.io/projected/38e569cd-e7c8-4a6c-a385-5a352a7e6b23-kube-api-access-s7zt9\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt\" (UID: \"38e569cd-e7c8-4a6c-a385-5a352a7e6b23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt"
Nov 03 08:23:01.870156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:01.870131    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt"
Nov 03 08:23:02.156641 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:23:02.156609    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e569cd_e7c8_4a6c_a385_5a352a7e6b23.slice/crio-18028112ed54d8d8a7c54745d872c2d89cde6d31b0a94b71f6fecb83de302b9a WatchSource:0}: Error finding container 18028112ed54d8d8a7c54745d872c2d89cde6d31b0a94b71f6fecb83de302b9a: Status 404 returned error can't find the container with id 18028112ed54d8d8a7c54745d872c2d89cde6d31b0a94b71f6fecb83de302b9a
Nov 03 08:23:02.157117 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:02.156801    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt"]
Nov 03 08:23:02.802256 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:02.802215    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt" event={"ID":"38e569cd-e7c8-4a6c-a385-5a352a7e6b23","Type":"ContainerStarted","Data":"401859e7ded8ab3ca1cd8bf947e33c6c596078f905eb2dbcebe14c491e02dc4c"}
Nov 03 08:23:02.802438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:02.802319    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt" event={"ID":"38e569cd-e7c8-4a6c-a385-5a352a7e6b23","Type":"ContainerStarted","Data":"18028112ed54d8d8a7c54745d872c2d89cde6d31b0a94b71f6fecb83de302b9a"}
Nov 03 08:23:02.814727 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:02.814686    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt" podStartSLOduration=1.814674678 podStartE2EDuration="1.814674678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:23:02.813814461 +0000 UTC m=+1780.928278391" watchObservedRunningTime="2026-11-03 08:23:02.814674678 +0000 UTC m=+1780.929138613"
Nov 03 08:23:03.813849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:03.813812    5070 generic.go:356] "Generic (PLEG): container finished" podID="38e569cd-e7c8-4a6c-a385-5a352a7e6b23" containerID="401859e7ded8ab3ca1cd8bf947e33c6c596078f905eb2dbcebe14c491e02dc4c" exitCode=0
Nov 03 08:23:03.814305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:03.813875    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt" event={"ID":"38e569cd-e7c8-4a6c-a385-5a352a7e6b23","Type":"ContainerDied","Data":"401859e7ded8ab3ca1cd8bf947e33c6c596078f905eb2dbcebe14c491e02dc4c"}
Nov 03 08:23:05.271853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:05.271813    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw"]
Nov 03 08:23:05.287088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:05.287055    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt"
Nov 03 08:23:05.327864 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:05.327826    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-e81ace-pxlmw"]
Nov 03 08:23:05.344023 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:05.343992    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e569cd-e7c8-4a6c-a385-5a352a7e6b23-kube-api-access-s7zt9\" (UniqueName: \"kubernetes.io/projected/38e569cd-e7c8-4a6c-a385-5a352a7e6b23-kube-api-access-s7zt9\") pod \"38e569cd-e7c8-4a6c-a385-5a352a7e6b23\" (UID: \"38e569cd-e7c8-4a6c-a385-5a352a7e6b23\") "
Nov 03 08:23:05.344392 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:05.344371    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38e569cd-e7c8-4a6c-a385-5a352a7e6b23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e569cd-e7c8-4a6c-a385-5a352a7e6b23-maas-api-xks-ca\") pod \"38e569cd-e7c8-4a6c-a385-5a352a7e6b23\" (UID: \"38e569cd-e7c8-4a6c-a385-5a352a7e6b23\") "
Nov 03 08:23:05.344554 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:05.344537    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38e569cd-e7c8-4a6c-a385-5a352a7e6b23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e569cd-e7c8-4a6c-a385-5a352a7e6b23-maas-api-openshift-ca\") pod \"38e569cd-e7c8-4a6c-a385-5a352a7e6b23\" (UID: \"38e569cd-e7c8-4a6c-a385-5a352a7e6b23\") "
Nov 03 08:23:05.433790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:05.429792    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e569cd-e7c8-4a6c-a385-5a352a7e6b23-kube-api-access-s7zt9" pod "38e569cd-e7c8-4a6c-a385-5a352a7e6b23" (UID: "38e569cd-e7c8-4a6c-a385-5a352a7e6b23"). InnerVolumeSpecName "kube-api-access-s7zt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:05.449793 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:05.440011    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e569cd-e7c8-4a6c-a385-5a352a7e6b23-maas-api-openshift-ca" pod "38e569cd-e7c8-4a6c-a385-5a352a7e6b23" (UID: "38e569cd-e7c8-4a6c-a385-5a352a7e6b23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:05.451914 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:05.451878    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7zt9\" (UniqueName: \"kubernetes.io/projected/38e569cd-e7c8-4a6c-a385-5a352a7e6b23-kube-api-access-s7zt9\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:05.452111 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:05.452096    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e569cd-e7c8-4a6c-a385-5a352a7e6b23-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:05.499830 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:05.486216    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38e569cd-e7c8-4a6c-a385-5a352a7e6b23-maas-api-xks-ca" pod "38e569cd-e7c8-4a6c-a385-5a352a7e6b23" (UID: "38e569cd-e7c8-4a6c-a385-5a352a7e6b23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:05.554195 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:05.554158    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e569cd-e7c8-4a6c-a385-5a352a7e6b23-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:05.842318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:05.842242    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt"
Nov 03 08:23:05.842318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:05.842274    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt" event={"ID":"38e569cd-e7c8-4a6c-a385-5a352a7e6b23","Type":"ContainerDied","Data":"18028112ed54d8d8a7c54745d872c2d89cde6d31b0a94b71f6fecb83de302b9a"}
Nov 03 08:23:05.842506 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:05.842320    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18028112ed54d8d8a7c54745d872c2d89cde6d31b0a94b71f6fecb83de302b9a"
Nov 03 08:23:06.029398 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.029370    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6" path="/var/lib/kubelet/pods/f53bc5e7-aae7-46f4-b95f-8ed3c60ec4d6/volumes"
Nov 03 08:23:06.224266 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.224237    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:23:06.270156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.270124    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08fbd019-0105-4baf-8c74-ae606e368876-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08fbd019-0105-4baf-8c74-ae606e368876-istiod-ca-cert\") pod \"08fbd019-0105-4baf-8c74-ae606e368876\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") "
Nov 03 08:23:06.270337 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.270213    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-workload-socket\") pod \"08fbd019-0105-4baf-8c74-ae606e368876\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") "
Nov 03 08:23:06.270411 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.270352    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08fbd019-0105-4baf-8c74-ae606e368876-kube-api-access-j6s9x\" (UniqueName: \"kubernetes.io/projected/08fbd019-0105-4baf-8c74-ae606e368876-kube-api-access-j6s9x\") pod \"08fbd019-0105-4baf-8c74-ae606e368876\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") "
Nov 03 08:23:06.271447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.270456    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-istio-data\") pod \"08fbd019-0105-4baf-8c74-ae606e368876\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") "
Nov 03 08:23:06.271447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.270490    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08fbd019-0105-4baf-8c74-ae606e368876-istio-token\" (UniqueName: \"kubernetes.io/projected/08fbd019-0105-4baf-8c74-ae606e368876-istio-token\") pod \"08fbd019-0105-4baf-8c74-ae606e368876\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") "
Nov 03 08:23:06.271447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.270548    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-istio-envoy\") pod \"08fbd019-0105-4baf-8c74-ae606e368876\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") "
Nov 03 08:23:06.271447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.270595    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08fbd019-0105-4baf-8c74-ae606e368876-istiod-ca-cert" pod "08fbd019-0105-4baf-8c74-ae606e368876" (UID: "08fbd019-0105-4baf-8c74-ae606e368876"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:06.271447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.270689    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/08fbd019-0105-4baf-8c74-ae606e368876-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08fbd019-0105-4baf-8c74-ae606e368876-istio-podinfo\") pod \"08fbd019-0105-4baf-8c74-ae606e368876\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") "
Nov 03 08:23:06.271447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.270724    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-workload-certs\") pod \"08fbd019-0105-4baf-8c74-ae606e368876\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") "
Nov 03 08:23:06.271447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.270839    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-credential-socket\") pod \"08fbd019-0105-4baf-8c74-ae606e368876\" (UID: \"08fbd019-0105-4baf-8c74-ae606e368876\") "
Nov 03 08:23:06.271447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.270864    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-istio-data" pod "08fbd019-0105-4baf-8c74-ae606e368876" (UID: "08fbd019-0105-4baf-8c74-ae606e368876"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:06.271447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.271127    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-workload-certs" pod "08fbd019-0105-4baf-8c74-ae606e368876" (UID: "08fbd019-0105-4baf-8c74-ae606e368876"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:06.271447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.271441    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-credential-socket" pod "08fbd019-0105-4baf-8c74-ae606e368876" (UID: "08fbd019-0105-4baf-8c74-ae606e368876"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:06.272387 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.271676    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-workload-socket" pod "08fbd019-0105-4baf-8c74-ae606e368876" (UID: "08fbd019-0105-4baf-8c74-ae606e368876"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:06.272387 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.272079    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08fbd019-0105-4baf-8c74-ae606e368876-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:06.272387 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.272102    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:06.272387 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.272116    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:06.272387 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.272129    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:06.272387 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.272145    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:06.273748 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.273716    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08fbd019-0105-4baf-8c74-ae606e368876-istio-token" pod "08fbd019-0105-4baf-8c74-ae606e368876" (UID: "08fbd019-0105-4baf-8c74-ae606e368876"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:06.274078 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.274053    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08fbd019-0105-4baf-8c74-ae606e368876-istio-podinfo" pod "08fbd019-0105-4baf-8c74-ae606e368876" (UID: "08fbd019-0105-4baf-8c74-ae606e368876"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:23:06.274352 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.274330    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08fbd019-0105-4baf-8c74-ae606e368876-kube-api-access-j6s9x" pod "08fbd019-0105-4baf-8c74-ae606e368876" (UID: "08fbd019-0105-4baf-8c74-ae606e368876"). InnerVolumeSpecName "kube-api-access-j6s9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:06.276616 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.276594    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-istio-envoy" pod "08fbd019-0105-4baf-8c74-ae606e368876" (UID: "08fbd019-0105-4baf-8c74-ae606e368876"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:06.374135 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.374069    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6s9x\" (UniqueName: \"kubernetes.io/projected/08fbd019-0105-4baf-8c74-ae606e368876-kube-api-access-j6s9x\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:06.374135 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.374096    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08fbd019-0105-4baf-8c74-ae606e368876-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:06.374135 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.374109    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08fbd019-0105-4baf-8c74-ae606e368876-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:06.374135 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.374122    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08fbd019-0105-4baf-8c74-ae606e368876-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:06.897741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.896500    5070 generic.go:356] "Generic (PLEG): container finished" podID="08fbd019-0105-4baf-8c74-ae606e368876" containerID="05b4a6bb2fe35070a26bc0a59f20826af5b461072cdab1dea6b526d77dcfa536" exitCode=0
Nov 03 08:23:06.897741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.896645    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx" event={"ID":"08fbd019-0105-4baf-8c74-ae606e368876","Type":"ContainerDied","Data":"05b4a6bb2fe35070a26bc0a59f20826af5b461072cdab1dea6b526d77dcfa536"}
Nov 03 08:23:06.897741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.896685    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx" event={"ID":"08fbd019-0105-4baf-8c74-ae606e368876","Type":"ContainerDied","Data":"ae494f6c2d76665842de3920d4311cb147bf5250f6d7030f251cf051fc63524b"}
Nov 03 08:23:06.897741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.896705    5070 scope.go:122] "RemoveContainer" containerID="05b4a6bb2fe35070a26bc0a59f20826af5b461072cdab1dea6b526d77dcfa536"
Nov 03 08:23:06.897741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.896884    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"
Nov 03 08:23:06.991347 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.990821    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd"]
Nov 03 08:23:06.998740 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.998125    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-179c16-ncnkd"]
Nov 03 08:23:06.998858 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:06.998816    5070 scope.go:122] "RemoveContainer" containerID="ed709d0dfc6afb094a5c7e3f852e8991cd96e0de1ae4f9ba5f1eca9f0d764ba9"
Nov 03 08:23:07.002905 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:07.002870    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"]
Nov 03 08:23:07.006909 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:07.006876    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a60004-openshift-default-7d9c55b6bf-mzqnx"]
Nov 03 08:23:07.012086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:07.012069    5070 scope.go:122] "RemoveContainer" containerID="05b4a6bb2fe35070a26bc0a59f20826af5b461072cdab1dea6b526d77dcfa536"
Nov 03 08:23:07.012356 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:23:07.012337    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b4a6bb2fe35070a26bc0a59f20826af5b461072cdab1dea6b526d77dcfa536\": container with ID starting with 05b4a6bb2fe35070a26bc0a59f20826af5b461072cdab1dea6b526d77dcfa536 not found: ID does not exist" containerID="05b4a6bb2fe35070a26bc0a59f20826af5b461072cdab1dea6b526d77dcfa536"
Nov 03 08:23:07.012408 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:07.012365    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b4a6bb2fe35070a26bc0a59f20826af5b461072cdab1dea6b526d77dcfa536"} err="failed to get container status \"05b4a6bb2fe35070a26bc0a59f20826af5b461072cdab1dea6b526d77dcfa536\": rpc error: code = NotFound desc = could not find container \"05b4a6bb2fe35070a26bc0a59f20826af5b461072cdab1dea6b526d77dcfa536\": container with ID starting with 05b4a6bb2fe35070a26bc0a59f20826af5b461072cdab1dea6b526d77dcfa536 not found: ID does not exist"
Nov 03 08:23:07.012408 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:07.012383    5070 scope.go:122] "RemoveContainer" containerID="ed709d0dfc6afb094a5c7e3f852e8991cd96e0de1ae4f9ba5f1eca9f0d764ba9"
Nov 03 08:23:07.012604 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:23:07.012586    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed709d0dfc6afb094a5c7e3f852e8991cd96e0de1ae4f9ba5f1eca9f0d764ba9\": container with ID starting with ed709d0dfc6afb094a5c7e3f852e8991cd96e0de1ae4f9ba5f1eca9f0d764ba9 not found: ID does not exist" containerID="ed709d0dfc6afb094a5c7e3f852e8991cd96e0de1ae4f9ba5f1eca9f0d764ba9"
Nov 03 08:23:07.012653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:07.012610    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed709d0dfc6afb094a5c7e3f852e8991cd96e0de1ae4f9ba5f1eca9f0d764ba9"} err="failed to get container status \"ed709d0dfc6afb094a5c7e3f852e8991cd96e0de1ae4f9ba5f1eca9f0d764ba9\": rpc error: code = NotFound desc = could not find container \"ed709d0dfc6afb094a5c7e3f852e8991cd96e0de1ae4f9ba5f1eca9f0d764ba9\": container with ID starting with ed709d0dfc6afb094a5c7e3f852e8991cd96e0de1ae4f9ba5f1eca9f0d764ba9 not found: ID does not exist"
Nov 03 08:23:08.028458 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.028428    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08fbd019-0105-4baf-8c74-ae606e368876" path="/var/lib/kubelet/pods/08fbd019-0105-4baf-8c74-ae606e368876/volumes"
Nov 03 08:23:08.028978 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.028961    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b44fb15-f8cb-4bd2-91f8-9d784d1a0083" path="/var/lib/kubelet/pods/3b44fb15-f8cb-4bd2-91f8-9d784d1a0083/volumes"
Nov 03 08:23:08.739857 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.739821    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7"]
Nov 03 08:23:08.740682 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.740660    5070 cpu_manager.go:414] "containerMap: removing container" podUID="08fbd019-0105-4baf-8c74-ae606e368876" containerName="istio-proxy"
Nov 03 08:23:08.740682 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.740677    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08fbd019-0105-4baf-8c74-ae606e368876" containerName="istio-proxy"
Nov 03 08:23:08.740942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.740722    5070 cpu_manager.go:414] "containerMap: removing container" podUID="08fbd019-0105-4baf-8c74-ae606e368876" containerName="istio-proxy"
Nov 03 08:23:08.740942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.740727    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08fbd019-0105-4baf-8c74-ae606e368876" containerName="istio-proxy"
Nov 03 08:23:08.740942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.740741    5070 cpu_manager.go:414] "containerMap: removing container" podUID="38e569cd-e7c8-4a6c-a385-5a352a7e6b23" containerName="revoke-keys"
Nov 03 08:23:08.740942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.740748    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e569cd-e7c8-4a6c-a385-5a352a7e6b23" containerName="revoke-keys"
Nov 03 08:23:08.740942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.740877    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="08fbd019-0105-4baf-8c74-ae606e368876" containerName="istio-proxy"
Nov 03 08:23:08.740942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.740896    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e569cd-e7c8-4a6c-a385-5a352a7e6b23" containerName="revoke-keys"
Nov 03 08:23:08.744250 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.744231    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7"
Nov 03 08:23:08.748883 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.748837    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:23:08.749012 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.748980    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:23:08.749076 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.749019    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:23:08.749076 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.749067    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:23:08.752399 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.752370    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7"]
Nov 03 08:23:08.770670 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.770647    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce2843e0-e7f1-4e37-ba01-11599705c09e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09c9d667-wqmm7\" (UID: \"ce2843e0-e7f1-4e37-ba01-11599705c09e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7"
Nov 03 08:23:08.770829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.770802    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xmq2\" (UniqueName: \"kubernetes.io/projected/ce2843e0-e7f1-4e37-ba01-11599705c09e-kube-api-access-4xmq2\") pod \"maas-api-revoke-sub-09c9d667-wqmm7\" (UID: \"ce2843e0-e7f1-4e37-ba01-11599705c09e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7"
Nov 03 08:23:08.770905 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.770851    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce2843e0-e7f1-4e37-ba01-11599705c09e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09c9d667-wqmm7\" (UID: \"ce2843e0-e7f1-4e37-ba01-11599705c09e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7"
Nov 03 08:23:08.872369 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.872335    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce2843e0-e7f1-4e37-ba01-11599705c09e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09c9d667-wqmm7\" (UID: \"ce2843e0-e7f1-4e37-ba01-11599705c09e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7"
Nov 03 08:23:08.872495 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.872440    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xmq2\" (UniqueName: \"kubernetes.io/projected/ce2843e0-e7f1-4e37-ba01-11599705c09e-kube-api-access-4xmq2\") pod \"maas-api-revoke-sub-09c9d667-wqmm7\" (UID: \"ce2843e0-e7f1-4e37-ba01-11599705c09e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7"
Nov 03 08:23:08.872548 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.872497    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce2843e0-e7f1-4e37-ba01-11599705c09e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09c9d667-wqmm7\" (UID: \"ce2843e0-e7f1-4e37-ba01-11599705c09e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7"
Nov 03 08:23:08.873211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.873188    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce2843e0-e7f1-4e37-ba01-11599705c09e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09c9d667-wqmm7\" (UID: \"ce2843e0-e7f1-4e37-ba01-11599705c09e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7"
Nov 03 08:23:08.874910 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.874887    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce2843e0-e7f1-4e37-ba01-11599705c09e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09c9d667-wqmm7\" (UID: \"ce2843e0-e7f1-4e37-ba01-11599705c09e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7"
Nov 03 08:23:08.881972 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:08.881951    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xmq2\" (UniqueName: \"kubernetes.io/projected/ce2843e0-e7f1-4e37-ba01-11599705c09e-kube-api-access-4xmq2\") pod \"maas-api-revoke-sub-09c9d667-wqmm7\" (UID: \"ce2843e0-e7f1-4e37-ba01-11599705c09e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7"
Nov 03 08:23:09.047911 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:09.047840    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-042af12e-cgsh4"]
Nov 03 08:23:09.048766 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:09.048740    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="08fbd019-0105-4baf-8c74-ae606e368876" containerName="istio-proxy"
Nov 03 08:23:09.051400 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:09.051383    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-042af12e-cgsh4"
Nov 03 08:23:09.060677 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:09.060651    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-042af12e-cgsh4"]
Nov 03 08:23:09.066004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:09.065978    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7"
Nov 03 08:23:09.093626 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:09.093489    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304875a7-473f-4fbf-99ba-8548ac801eb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-042af12e-cgsh4\" (UID: \"304875a7-473f-4fbf-99ba-8548ac801eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-042af12e-cgsh4"
Nov 03 08:23:09.093626 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:09.093581    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8vjl\" (UniqueName: \"kubernetes.io/projected/304875a7-473f-4fbf-99ba-8548ac801eb5-kube-api-access-k8vjl\") pod \"maas-api-revoke-sub-042af12e-cgsh4\" (UID: \"304875a7-473f-4fbf-99ba-8548ac801eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-042af12e-cgsh4"
Nov 03 08:23:09.093988 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:09.093962    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304875a7-473f-4fbf-99ba-8548ac801eb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-042af12e-cgsh4\" (UID: \"304875a7-473f-4fbf-99ba-8548ac801eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-042af12e-cgsh4"
Nov 03 08:23:09.197802 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:09.195815    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304875a7-473f-4fbf-99ba-8548ac801eb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-042af12e-cgsh4\" (UID: \"304875a7-473f-4fbf-99ba-8548ac801eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-042af12e-cgsh4"
Nov 03 08:23:09.197802 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:09.195874    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8vjl\" (UniqueName: \"kubernetes.io/projected/304875a7-473f-4fbf-99ba-8548ac801eb5-kube-api-access-k8vjl\") pod \"maas-api-revoke-sub-042af12e-cgsh4\" (UID: \"304875a7-473f-4fbf-99ba-8548ac801eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-042af12e-cgsh4"
Nov 03 08:23:09.197802 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:09.196032    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304875a7-473f-4fbf-99ba-8548ac801eb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-042af12e-cgsh4\" (UID: \"304875a7-473f-4fbf-99ba-8548ac801eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-042af12e-cgsh4"
Nov 03 08:23:09.197802 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:09.196771    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304875a7-473f-4fbf-99ba-8548ac801eb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-042af12e-cgsh4\" (UID: \"304875a7-473f-4fbf-99ba-8548ac801eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-042af12e-cgsh4"
Nov 03 08:23:09.200708 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:09.200662    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304875a7-473f-4fbf-99ba-8548ac801eb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-042af12e-cgsh4\" (UID: \"304875a7-473f-4fbf-99ba-8548ac801eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-042af12e-cgsh4"
Nov 03 08:23:09.204894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:09.204871    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8vjl\" (UniqueName: \"kubernetes.io/projected/304875a7-473f-4fbf-99ba-8548ac801eb5-kube-api-access-k8vjl\") pod \"maas-api-revoke-sub-042af12e-cgsh4\" (UID: \"304875a7-473f-4fbf-99ba-8548ac801eb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-042af12e-cgsh4"
Nov 03 08:23:09.355569 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:09.355540    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7"]
Nov 03 08:23:09.355699 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:23:09.355581    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2843e0_e7f1_4e37_ba01_11599705c09e.slice/crio-2a6dac4ac0a153666462d96c53aeb548dbc72490b15ee48746d0bcae369a5677 WatchSource:0}: Error finding container 2a6dac4ac0a153666462d96c53aeb548dbc72490b15ee48746d0bcae369a5677: Status 404 returned error can't find the container with id 2a6dac4ac0a153666462d96c53aeb548dbc72490b15ee48746d0bcae369a5677
Nov 03 08:23:09.369908 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:09.369885    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-042af12e-cgsh4"
Nov 03 08:23:09.657992 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:09.657969    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-042af12e-cgsh4"]
Nov 03 08:23:09.999418 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:09.999322    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7" event={"ID":"ce2843e0-e7f1-4e37-ba01-11599705c09e","Type":"ContainerStarted","Data":"6da8397c627411c72a29ac5d413c4d8f89a199960ca3faeb2eed0e382e8655d4"}
Nov 03 08:23:09.999418 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:09.999389    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7" event={"ID":"ce2843e0-e7f1-4e37-ba01-11599705c09e","Type":"ContainerStarted","Data":"2a6dac4ac0a153666462d96c53aeb548dbc72490b15ee48746d0bcae369a5677"}
Nov 03 08:23:10.000746 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:10.000722    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-042af12e-cgsh4" event={"ID":"304875a7-473f-4fbf-99ba-8548ac801eb5","Type":"ContainerStarted","Data":"fb42193bb4d11a43127cf9c16a8093d58eeb3373f183ac7f55a80bf3f301b7b7"}
Nov 03 08:23:10.016783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:10.016727    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7" podStartSLOduration=2.016714598 podStartE2EDuration="2.016714598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:23:10.012696735 +0000 UTC m=+1788.127160665" watchObservedRunningTime="2026-11-03 08:23:10.016714598 +0000 UTC m=+1788.131178529"
Nov 03 08:23:11.019131 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:11.019090    5070 generic.go:356] "Generic (PLEG): container finished" podID="ce2843e0-e7f1-4e37-ba01-11599705c09e" containerID="6da8397c627411c72a29ac5d413c4d8f89a199960ca3faeb2eed0e382e8655d4" exitCode=0
Nov 03 08:23:11.019626 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:11.019176    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7" event={"ID":"ce2843e0-e7f1-4e37-ba01-11599705c09e","Type":"ContainerDied","Data":"6da8397c627411c72a29ac5d413c4d8f89a199960ca3faeb2eed0e382e8655d4"}
Nov 03 08:23:11.021918 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:11.021888    5070 generic.go:356] "Generic (PLEG): container finished" podID="304875a7-473f-4fbf-99ba-8548ac801eb5" containerID="cd909319e40c089e64bfd448294f742abda0f3b7e5cca03cda81489c6f2cbc85" exitCode=0
Nov 03 08:23:11.022253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:11.021938    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-042af12e-cgsh4" event={"ID":"304875a7-473f-4fbf-99ba-8548ac801eb5","Type":"ContainerDied","Data":"cd909319e40c089e64bfd448294f742abda0f3b7e5cca03cda81489c6f2cbc85"}
Nov 03 08:23:12.660783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.656954    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7"
Nov 03 08:23:12.674792 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.662082    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-042af12e-cgsh4"
Nov 03 08:23:12.776909 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.776823    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce2843e0-e7f1-4e37-ba01-11599705c09e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce2843e0-e7f1-4e37-ba01-11599705c09e-maas-api-xks-ca\") pod \"ce2843e0-e7f1-4e37-ba01-11599705c09e\" (UID: \"ce2843e0-e7f1-4e37-ba01-11599705c09e\") "
Nov 03 08:23:12.777077 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.776967    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/304875a7-473f-4fbf-99ba-8548ac801eb5-kube-api-access-k8vjl\" (UniqueName: \"kubernetes.io/projected/304875a7-473f-4fbf-99ba-8548ac801eb5-kube-api-access-k8vjl\") pod \"304875a7-473f-4fbf-99ba-8548ac801eb5\" (UID: \"304875a7-473f-4fbf-99ba-8548ac801eb5\") "
Nov 03 08:23:12.777077 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.777023    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce2843e0-e7f1-4e37-ba01-11599705c09e-kube-api-access-4xmq2\" (UniqueName: \"kubernetes.io/projected/ce2843e0-e7f1-4e37-ba01-11599705c09e-kube-api-access-4xmq2\") pod \"ce2843e0-e7f1-4e37-ba01-11599705c09e\" (UID: \"ce2843e0-e7f1-4e37-ba01-11599705c09e\") "
Nov 03 08:23:12.777192 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.777108    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce2843e0-e7f1-4e37-ba01-11599705c09e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce2843e0-e7f1-4e37-ba01-11599705c09e-maas-api-openshift-ca\") pod \"ce2843e0-e7f1-4e37-ba01-11599705c09e\" (UID: \"ce2843e0-e7f1-4e37-ba01-11599705c09e\") "
Nov 03 08:23:12.777267 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.777250    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/304875a7-473f-4fbf-99ba-8548ac801eb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304875a7-473f-4fbf-99ba-8548ac801eb5-maas-api-openshift-ca\") pod \"304875a7-473f-4fbf-99ba-8548ac801eb5\" (UID: \"304875a7-473f-4fbf-99ba-8548ac801eb5\") "
Nov 03 08:23:12.777357 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.777340    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/304875a7-473f-4fbf-99ba-8548ac801eb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304875a7-473f-4fbf-99ba-8548ac801eb5-maas-api-xks-ca\") pod \"304875a7-473f-4fbf-99ba-8548ac801eb5\" (UID: \"304875a7-473f-4fbf-99ba-8548ac801eb5\") "
Nov 03 08:23:12.778336 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.778257    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce2843e0-e7f1-4e37-ba01-11599705c09e-maas-api-openshift-ca" pod "ce2843e0-e7f1-4e37-ba01-11599705c09e" (UID: "ce2843e0-e7f1-4e37-ba01-11599705c09e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:12.779426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.779400    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/304875a7-473f-4fbf-99ba-8548ac801eb5-maas-api-openshift-ca" pod "304875a7-473f-4fbf-99ba-8548ac801eb5" (UID: "304875a7-473f-4fbf-99ba-8548ac801eb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:12.810783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.808540    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"]
Nov 03 08:23:12.810783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.808865    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f" podUID="157b7ff4-503c-4ecd-908c-2c078e875322" containerName="istio-proxy" containerID="cri-o://64d96e4cc0fa351fa91788e6f817bb055b1d68119a26fe761ab5ae4859c830cf" gracePeriod=30
Nov 03 08:23:12.835921 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.832700    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/304875a7-473f-4fbf-99ba-8548ac801eb5-maas-api-xks-ca" pod "304875a7-473f-4fbf-99ba-8548ac801eb5" (UID: "304875a7-473f-4fbf-99ba-8548ac801eb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:12.835921 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.834939    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce2843e0-e7f1-4e37-ba01-11599705c09e-maas-api-xks-ca" pod "ce2843e0-e7f1-4e37-ba01-11599705c09e" (UID: "ce2843e0-e7f1-4e37-ba01-11599705c09e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:12.835921 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.834992    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/304875a7-473f-4fbf-99ba-8548ac801eb5-kube-api-access-k8vjl" pod "304875a7-473f-4fbf-99ba-8548ac801eb5" (UID: "304875a7-473f-4fbf-99ba-8548ac801eb5"). InnerVolumeSpecName "kube-api-access-k8vjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:12.839219 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.836374    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2843e0-e7f1-4e37-ba01-11599705c09e-kube-api-access-4xmq2" pod "ce2843e0-e7f1-4e37-ba01-11599705c09e" (UID: "ce2843e0-e7f1-4e37-ba01-11599705c09e"). InnerVolumeSpecName "kube-api-access-4xmq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:12.883918 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.883871    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce2843e0-e7f1-4e37-ba01-11599705c09e-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:12.883918 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.883902    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8vjl\" (UniqueName: \"kubernetes.io/projected/304875a7-473f-4fbf-99ba-8548ac801eb5-kube-api-access-k8vjl\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:12.883918 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.883919    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xmq2\" (UniqueName: \"kubernetes.io/projected/ce2843e0-e7f1-4e37-ba01-11599705c09e-kube-api-access-4xmq2\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:12.884329 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.883934    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce2843e0-e7f1-4e37-ba01-11599705c09e-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:12.884329 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.883947    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/304875a7-473f-4fbf-99ba-8548ac801eb5-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:12.884329 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:12.883960    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/304875a7-473f-4fbf-99ba-8548ac801eb5-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:13.356817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.352312    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7" event={"ID":"ce2843e0-e7f1-4e37-ba01-11599705c09e","Type":"ContainerDied","Data":"2a6dac4ac0a153666462d96c53aeb548dbc72490b15ee48746d0bcae369a5677"}
Nov 03 08:23:13.356817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.352439    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a6dac4ac0a153666462d96c53aeb548dbc72490b15ee48746d0bcae369a5677"
Nov 03 08:23:13.356817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.352531    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7"
Nov 03 08:23:13.385318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.385281    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-042af12e-cgsh4" event={"ID":"304875a7-473f-4fbf-99ba-8548ac801eb5","Type":"ContainerDied","Data":"fb42193bb4d11a43127cf9c16a8093d58eeb3373f183ac7f55a80bf3f301b7b7"}
Nov 03 08:23:13.385532 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.385518    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb42193bb4d11a43127cf9c16a8093d58eeb3373f183ac7f55a80bf3f301b7b7"
Nov 03 08:23:13.385692 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.385681    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-042af12e-cgsh4"
Nov 03 08:23:13.394433 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.389344    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt"]
Nov 03 08:23:13.396642 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.396616    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-fe584a-265mt"]
Nov 03 08:23:13.542271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.542242    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48"]
Nov 03 08:23:13.543075 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.543060    5070 cpu_manager.go:414] "containerMap: removing container" podUID="304875a7-473f-4fbf-99ba-8548ac801eb5" containerName="revoke-keys"
Nov 03 08:23:13.543119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.543077    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="304875a7-473f-4fbf-99ba-8548ac801eb5" containerName="revoke-keys"
Nov 03 08:23:13.543119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.543104    5070 cpu_manager.go:414] "containerMap: removing container" podUID="ce2843e0-e7f1-4e37-ba01-11599705c09e" containerName="revoke-keys"
Nov 03 08:23:13.543119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.543109    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2843e0-e7f1-4e37-ba01-11599705c09e" containerName="revoke-keys"
Nov 03 08:23:13.543253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.543244    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="304875a7-473f-4fbf-99ba-8548ac801eb5" containerName="revoke-keys"
Nov 03 08:23:13.543292 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.543258    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2843e0-e7f1-4e37-ba01-11599705c09e" containerName="revoke-keys"
Nov 03 08:23:13.547029 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.547011    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48"
Nov 03 08:23:13.551293 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.551267    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:23:13.551293 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.551280    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:23:13.551507 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.551354    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:23:13.551507 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.551399    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:23:13.556986 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.556964    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48"]
Nov 03 08:23:13.594647 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.594619    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b57c27-f0e7-4864-b9f1-fa221b7d7bec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48\" (UID: \"61b57c27-f0e7-4864-b9f1-fa221b7d7bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48"
Nov 03 08:23:13.594806 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.594791    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b57c27-f0e7-4864-b9f1-fa221b7d7bec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48\" (UID: \"61b57c27-f0e7-4864-b9f1-fa221b7d7bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48"
Nov 03 08:23:13.594851 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.594833    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64qvt\" (UniqueName: \"kubernetes.io/projected/61b57c27-f0e7-4864-b9f1-fa221b7d7bec-kube-api-access-64qvt\") pod \"maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48\" (UID: \"61b57c27-f0e7-4864-b9f1-fa221b7d7bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48"
Nov 03 08:23:13.702835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.701988    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b57c27-f0e7-4864-b9f1-fa221b7d7bec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48\" (UID: \"61b57c27-f0e7-4864-b9f1-fa221b7d7bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48"
Nov 03 08:23:13.702835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.702070    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64qvt\" (UniqueName: \"kubernetes.io/projected/61b57c27-f0e7-4864-b9f1-fa221b7d7bec-kube-api-access-64qvt\") pod \"maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48\" (UID: \"61b57c27-f0e7-4864-b9f1-fa221b7d7bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48"
Nov 03 08:23:13.702835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.702374    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b57c27-f0e7-4864-b9f1-fa221b7d7bec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48\" (UID: \"61b57c27-f0e7-4864-b9f1-fa221b7d7bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48"
Nov 03 08:23:13.714069 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.712546    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b57c27-f0e7-4864-b9f1-fa221b7d7bec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48\" (UID: \"61b57c27-f0e7-4864-b9f1-fa221b7d7bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48"
Nov 03 08:23:13.719983 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.716391    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b57c27-f0e7-4864-b9f1-fa221b7d7bec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48\" (UID: \"61b57c27-f0e7-4864-b9f1-fa221b7d7bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48"
Nov 03 08:23:13.723583 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.723556    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64qvt\" (UniqueName: \"kubernetes.io/projected/61b57c27-f0e7-4864-b9f1-fa221b7d7bec-kube-api-access-64qvt\") pod \"maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48\" (UID: \"61b57c27-f0e7-4864-b9f1-fa221b7d7bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48"
Nov 03 08:23:13.867214 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:13.867183    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48"
Nov 03 08:23:14.028753 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:14.028684    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38e569cd-e7c8-4a6c-a385-5a352a7e6b23" path="/var/lib/kubelet/pods/38e569cd-e7c8-4a6c-a385-5a352a7e6b23/volumes"
Nov 03 08:23:14.115587 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:14.115558    5070 scope.go:122] "RemoveContainer" containerID="aeab18f2416d23fc7f4c44c3bc5a7381d990c8c3f40204fbc8b7e38835f8a4dc"
Nov 03 08:23:14.160635 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:23:14.160602    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b57c27_f0e7_4864_b9f1_fa221b7d7bec.slice/crio-6a04eb0f9a1e0d9bde9ce94b26fee29edcfd42d666797aa8c543acae4b6c8690 WatchSource:0}: Error finding container 6a04eb0f9a1e0d9bde9ce94b26fee29edcfd42d666797aa8c543acae4b6c8690: Status 404 returned error can't find the container with id 6a04eb0f9a1e0d9bde9ce94b26fee29edcfd42d666797aa8c543acae4b6c8690
Nov 03 08:23:14.162479 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:14.162453    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48"]
Nov 03 08:23:14.396209 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:14.396182    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48" event={"ID":"61b57c27-f0e7-4864-b9f1-fa221b7d7bec","Type":"ContainerStarted","Data":"6a04eb0f9a1e0d9bde9ce94b26fee29edcfd42d666797aa8c543acae4b6c8690"}
Nov 03 08:23:15.408656 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:15.408613    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48" event={"ID":"61b57c27-f0e7-4864-b9f1-fa221b7d7bec","Type":"ContainerStarted","Data":"621bd062dbfa2978c0edf06adaac5a1a0a4cba518c3028e4fd96698d60662f1e"}
Nov 03 08:23:15.422549 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:15.422508    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48" podStartSLOduration=2.422495699 podStartE2EDuration="2.422495699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:23:15.422120568 +0000 UTC m=+1793.536584478" watchObservedRunningTime="2026-11-03 08:23:15.422495699 +0000 UTC m=+1793.536959630"
Nov 03 08:23:16.420420 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:16.420348    5070 generic.go:356] "Generic (PLEG): container finished" podID="61b57c27-f0e7-4864-b9f1-fa221b7d7bec" containerID="621bd062dbfa2978c0edf06adaac5a1a0a4cba518c3028e4fd96698d60662f1e" exitCode=0
Nov 03 08:23:16.420991 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:16.420500    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48" event={"ID":"61b57c27-f0e7-4864-b9f1-fa221b7d7bec","Type":"ContainerDied","Data":"621bd062dbfa2978c0edf06adaac5a1a0a4cba518c3028e4fd96698d60662f1e"}
Nov 03 08:23:17.781789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:17.781699    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48"
Nov 03 08:23:17.851613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:17.851583    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61b57c27-f0e7-4864-b9f1-fa221b7d7bec-kube-api-access-64qvt\" (UniqueName: \"kubernetes.io/projected/61b57c27-f0e7-4864-b9f1-fa221b7d7bec-kube-api-access-64qvt\") pod \"61b57c27-f0e7-4864-b9f1-fa221b7d7bec\" (UID: \"61b57c27-f0e7-4864-b9f1-fa221b7d7bec\") "
Nov 03 08:23:17.851833 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:17.851814    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61b57c27-f0e7-4864-b9f1-fa221b7d7bec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b57c27-f0e7-4864-b9f1-fa221b7d7bec-maas-api-xks-ca\") pod \"61b57c27-f0e7-4864-b9f1-fa221b7d7bec\" (UID: \"61b57c27-f0e7-4864-b9f1-fa221b7d7bec\") "
Nov 03 08:23:17.851926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:17.851896    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61b57c27-f0e7-4864-b9f1-fa221b7d7bec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b57c27-f0e7-4864-b9f1-fa221b7d7bec-maas-api-openshift-ca\") pod \"61b57c27-f0e7-4864-b9f1-fa221b7d7bec\" (UID: \"61b57c27-f0e7-4864-b9f1-fa221b7d7bec\") "
Nov 03 08:23:17.852278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:17.852247    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61b57c27-f0e7-4864-b9f1-fa221b7d7bec-maas-api-openshift-ca" pod "61b57c27-f0e7-4864-b9f1-fa221b7d7bec" (UID: "61b57c27-f0e7-4864-b9f1-fa221b7d7bec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:17.852902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:17.852866    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61b57c27-f0e7-4864-b9f1-fa221b7d7bec-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:17.854353 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:17.854327    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61b57c27-f0e7-4864-b9f1-fa221b7d7bec-kube-api-access-64qvt" pod "61b57c27-f0e7-4864-b9f1-fa221b7d7bec" (UID: "61b57c27-f0e7-4864-b9f1-fa221b7d7bec"). InnerVolumeSpecName "kube-api-access-64qvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:17.856128 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:17.856097    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61b57c27-f0e7-4864-b9f1-fa221b7d7bec-maas-api-xks-ca" pod "61b57c27-f0e7-4864-b9f1-fa221b7d7bec" (UID: "61b57c27-f0e7-4864-b9f1-fa221b7d7bec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:17.955182 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:17.955155    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61b57c27-f0e7-4864-b9f1-fa221b7d7bec-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:17.955292 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:17.955187    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64qvt\" (UniqueName: \"kubernetes.io/projected/61b57c27-f0e7-4864-b9f1-fa221b7d7bec-kube-api-access-64qvt\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:18.057145 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.054896    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"]
Nov 03 08:23:18.067531 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.067505    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"]
Nov 03 08:23:18.134109 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.134086    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:23:18.160053 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.160022    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-istio-data\") pod \"157b7ff4-503c-4ecd-908c-2c078e875322\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") "
Nov 03 08:23:18.160193 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.160111    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/157b7ff4-503c-4ecd-908c-2c078e875322-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157b7ff4-503c-4ecd-908c-2c078e875322-istio-podinfo\") pod \"157b7ff4-503c-4ecd-908c-2c078e875322\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") "
Nov 03 08:23:18.160193 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.160172    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/157b7ff4-503c-4ecd-908c-2c078e875322-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157b7ff4-503c-4ecd-908c-2c078e875322-istiod-ca-cert\") pod \"157b7ff4-503c-4ecd-908c-2c078e875322\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") "
Nov 03 08:23:18.160317 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.160267    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-istio-data" pod "157b7ff4-503c-4ecd-908c-2c078e875322" (UID: "157b7ff4-503c-4ecd-908c-2c078e875322"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:18.160317 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.160308    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157b7ff4-503c-4ecd-908c-2c078e875322-kube-api-access-n2tp6\" (UniqueName: \"kubernetes.io/projected/157b7ff4-503c-4ecd-908c-2c078e875322-kube-api-access-n2tp6\") pod \"157b7ff4-503c-4ecd-908c-2c078e875322\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") "
Nov 03 08:23:18.160425 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.160344    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157b7ff4-503c-4ecd-908c-2c078e875322-istio-token\" (UniqueName: \"kubernetes.io/projected/157b7ff4-503c-4ecd-908c-2c078e875322-istio-token\") pod \"157b7ff4-503c-4ecd-908c-2c078e875322\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") "
Nov 03 08:23:18.160491 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.160445    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-workload-socket\") pod \"157b7ff4-503c-4ecd-908c-2c078e875322\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") "
Nov 03 08:23:18.160491 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.160483    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-istio-envoy\") pod \"157b7ff4-503c-4ecd-908c-2c078e875322\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") "
Nov 03 08:23:18.160653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.160621    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/157b7ff4-503c-4ecd-908c-2c078e875322-istiod-ca-cert" pod "157b7ff4-503c-4ecd-908c-2c078e875322" (UID: "157b7ff4-503c-4ecd-908c-2c078e875322"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:18.160710 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.160685    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-workload-certs\") pod \"157b7ff4-503c-4ecd-908c-2c078e875322\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") "
Nov 03 08:23:18.160811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.160797    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-credential-socket\") pod \"157b7ff4-503c-4ecd-908c-2c078e875322\" (UID: \"157b7ff4-503c-4ecd-908c-2c078e875322\") "
Nov 03 08:23:18.160921 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.160897    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-workload-socket" pod "157b7ff4-503c-4ecd-908c-2c078e875322" (UID: "157b7ff4-503c-4ecd-908c-2c078e875322"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:18.161199 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.161172    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-workload-certs" pod "157b7ff4-503c-4ecd-908c-2c078e875322" (UID: "157b7ff4-503c-4ecd-908c-2c078e875322"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:18.161316 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.161200    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-credential-socket" pod "157b7ff4-503c-4ecd-908c-2c078e875322" (UID: "157b7ff4-503c-4ecd-908c-2c078e875322"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:18.161622 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.161598    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:18.161697 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.161627    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:18.161697 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.161644    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:18.161697 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.161660    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:18.161697 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.161675    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/157b7ff4-503c-4ecd-908c-2c078e875322-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:18.181677 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.180774    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157b7ff4-503c-4ecd-908c-2c078e875322-istio-token" pod "157b7ff4-503c-4ecd-908c-2c078e875322" (UID: "157b7ff4-503c-4ecd-908c-2c078e875322"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:18.181677 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.181438    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157b7ff4-503c-4ecd-908c-2c078e875322-kube-api-access-n2tp6" pod "157b7ff4-503c-4ecd-908c-2c078e875322" (UID: "157b7ff4-503c-4ecd-908c-2c078e875322"). InnerVolumeSpecName "kube-api-access-n2tp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:18.181677 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.181635    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/157b7ff4-503c-4ecd-908c-2c078e875322-istio-podinfo" pod "157b7ff4-503c-4ecd-908c-2c078e875322" (UID: "157b7ff4-503c-4ecd-908c-2c078e875322"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:23:18.181953 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.181921    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-istio-envoy" pod "157b7ff4-503c-4ecd-908c-2c078e875322" (UID: "157b7ff4-503c-4ecd-908c-2c078e875322"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:18.263557 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.263529    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2tp6\" (UniqueName: \"kubernetes.io/projected/157b7ff4-503c-4ecd-908c-2c078e875322-kube-api-access-n2tp6\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:18.263557 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.263555    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/157b7ff4-503c-4ecd-908c-2c078e875322-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:18.263829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.263572    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/157b7ff4-503c-4ecd-908c-2c078e875322-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:18.263829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.263583    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/157b7ff4-503c-4ecd-908c-2c078e875322-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:18.565409 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.564543    5070 generic.go:356] "Generic (PLEG): container finished" podID="157b7ff4-503c-4ecd-908c-2c078e875322" containerID="64d96e4cc0fa351fa91788e6f817bb055b1d68119a26fe761ab5ae4859c830cf" exitCode=0
Nov 03 08:23:18.565409 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.564638    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"
Nov 03 08:23:18.565409 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.564733    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f" event={"ID":"157b7ff4-503c-4ecd-908c-2c078e875322","Type":"ContainerDied","Data":"64d96e4cc0fa351fa91788e6f817bb055b1d68119a26fe761ab5ae4859c830cf"}
Nov 03 08:23:18.565409 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.564855    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f" event={"ID":"157b7ff4-503c-4ecd-908c-2c078e875322","Type":"ContainerDied","Data":"3274b40ffdfa693d42bf9784291f4e0532ea21f40fcecffb9ad28f309ea9def0"}
Nov 03 08:23:18.565409 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.564894    5070 scope.go:122] "RemoveContainer" containerID="64d96e4cc0fa351fa91788e6f817bb055b1d68119a26fe761ab5ae4859c830cf"
Nov 03 08:23:18.574126 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.573586    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48" event={"ID":"61b57c27-f0e7-4864-b9f1-fa221b7d7bec","Type":"ContainerDied","Data":"6a04eb0f9a1e0d9bde9ce94b26fee29edcfd42d666797aa8c543acae4b6c8690"}
Nov 03 08:23:18.574126 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.573650    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a04eb0f9a1e0d9bde9ce94b26fee29edcfd42d666797aa8c543acae4b6c8690"
Nov 03 08:23:18.574126 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.573727    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48"
Nov 03 08:23:18.589534 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.589508    5070 scope.go:122] "RemoveContainer" containerID="355dd9ad7c20d3e2267799e06e05f65b9b59d7b0d068944ac3522476d43805a9"
Nov 03 08:23:18.601955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.601931    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"]
Nov 03 08:23:18.606200 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.606166    5070 scope.go:122] "RemoveContainer" containerID="64d96e4cc0fa351fa91788e6f817bb055b1d68119a26fe761ab5ae4859c830cf"
Nov 03 08:23:18.606512 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:23:18.606483    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64d96e4cc0fa351fa91788e6f817bb055b1d68119a26fe761ab5ae4859c830cf\": container with ID starting with 64d96e4cc0fa351fa91788e6f817bb055b1d68119a26fe761ab5ae4859c830cf not found: ID does not exist" containerID="64d96e4cc0fa351fa91788e6f817bb055b1d68119a26fe761ab5ae4859c830cf"
Nov 03 08:23:18.606601 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.606522    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64d96e4cc0fa351fa91788e6f817bb055b1d68119a26fe761ab5ae4859c830cf"} err="failed to get container status \"64d96e4cc0fa351fa91788e6f817bb055b1d68119a26fe761ab5ae4859c830cf\": rpc error: code = NotFound desc = could not find container \"64d96e4cc0fa351fa91788e6f817bb055b1d68119a26fe761ab5ae4859c830cf\": container with ID starting with 64d96e4cc0fa351fa91788e6f817bb055b1d68119a26fe761ab5ae4859c830cf not found: ID does not exist"
Nov 03 08:23:18.606601 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.606549    5070 scope.go:122] "RemoveContainer" containerID="355dd9ad7c20d3e2267799e06e05f65b9b59d7b0d068944ac3522476d43805a9"
Nov 03 08:23:18.606864 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:23:18.606844    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"355dd9ad7c20d3e2267799e06e05f65b9b59d7b0d068944ac3522476d43805a9\": container with ID starting with 355dd9ad7c20d3e2267799e06e05f65b9b59d7b0d068944ac3522476d43805a9 not found: ID does not exist" containerID="355dd9ad7c20d3e2267799e06e05f65b9b59d7b0d068944ac3522476d43805a9"
Nov 03 08:23:18.606939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.606870    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"355dd9ad7c20d3e2267799e06e05f65b9b59d7b0d068944ac3522476d43805a9"} err="failed to get container status \"355dd9ad7c20d3e2267799e06e05f65b9b59d7b0d068944ac3522476d43805a9\": rpc error: code = NotFound desc = could not find container \"355dd9ad7c20d3e2267799e06e05f65b9b59d7b0d068944ac3522476d43805a9\": container with ID starting with 355dd9ad7c20d3e2267799e06e05f65b9b59d7b0d068944ac3522476d43805a9 not found: ID does not exist"
Nov 03 08:23:18.609425 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:18.609405    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-e81ace-openshift-default-668bddd8d7-64f4f"]
Nov 03 08:23:20.030153 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:20.030117    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="157b7ff4-503c-4ecd-908c-2c078e875322" path="/var/lib/kubelet/pods/157b7ff4-503c-4ecd-908c-2c078e875322/volumes"
Nov 03 08:23:21.000419 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.000386    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv"]
Nov 03 08:23:21.001232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.001214    5070 cpu_manager.go:414] "containerMap: removing container" podUID="157b7ff4-503c-4ecd-908c-2c078e875322" containerName="istio-proxy"
Nov 03 08:23:21.001232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.001232    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="157b7ff4-503c-4ecd-908c-2c078e875322" containerName="istio-proxy"
Nov 03 08:23:21.001807 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.001246    5070 cpu_manager.go:414] "containerMap: removing container" podUID="157b7ff4-503c-4ecd-908c-2c078e875322" containerName="istio-proxy"
Nov 03 08:23:21.001807 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.001251    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="157b7ff4-503c-4ecd-908c-2c078e875322" containerName="istio-proxy"
Nov 03 08:23:21.001807 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.001273    5070 cpu_manager.go:414] "containerMap: removing container" podUID="61b57c27-f0e7-4864-b9f1-fa221b7d7bec" containerName="revoke-keys"
Nov 03 08:23:21.001807 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.001278    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61b57c27-f0e7-4864-b9f1-fa221b7d7bec" containerName="revoke-keys"
Nov 03 08:23:21.001807 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.001389    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="61b57c27-f0e7-4864-b9f1-fa221b7d7bec" containerName="revoke-keys"
Nov 03 08:23:21.001807 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.001405    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="157b7ff4-503c-4ecd-908c-2c078e875322" containerName="istio-proxy"
Nov 03 08:23:21.001807 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.001415    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="157b7ff4-503c-4ecd-908c-2c078e875322" containerName="istio-proxy"
Nov 03 08:23:21.007013 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.006996    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv"
Nov 03 08:23:21.009993 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.009969    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:23:21.010462 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.010186    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:23:21.015121 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.015100    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv"]
Nov 03 08:23:21.039058 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.039038    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f195a785-22bc-46c1-833b-f63a37a2de85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv\" (UID: \"f195a785-22bc-46c1-833b-f63a37a2de85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv"
Nov 03 08:23:21.039410 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.039081    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlkmz\" (UniqueName: \"kubernetes.io/projected/f195a785-22bc-46c1-833b-f63a37a2de85-kube-api-access-dlkmz\") pod \"maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv\" (UID: \"f195a785-22bc-46c1-833b-f63a37a2de85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv"
Nov 03 08:23:21.039410 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.039277    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f195a785-22bc-46c1-833b-f63a37a2de85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv\" (UID: \"f195a785-22bc-46c1-833b-f63a37a2de85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv"
Nov 03 08:23:21.040398 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.040378    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:23:21.040492 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.040458    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:23:21.140900 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.140875    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f195a785-22bc-46c1-833b-f63a37a2de85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv\" (UID: \"f195a785-22bc-46c1-833b-f63a37a2de85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv"
Nov 03 08:23:21.141102 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.141085    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f195a785-22bc-46c1-833b-f63a37a2de85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv\" (UID: \"f195a785-22bc-46c1-833b-f63a37a2de85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv"
Nov 03 08:23:21.141156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.141127    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlkmz\" (UniqueName: \"kubernetes.io/projected/f195a785-22bc-46c1-833b-f63a37a2de85-kube-api-access-dlkmz\") pod \"maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv\" (UID: \"f195a785-22bc-46c1-833b-f63a37a2de85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv"
Nov 03 08:23:21.141503 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.141483    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f195a785-22bc-46c1-833b-f63a37a2de85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv\" (UID: \"f195a785-22bc-46c1-833b-f63a37a2de85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv"
Nov 03 08:23:21.143642 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.143620    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f195a785-22bc-46c1-833b-f63a37a2de85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv\" (UID: \"f195a785-22bc-46c1-833b-f63a37a2de85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv"
Nov 03 08:23:21.178496 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.178462    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlkmz\" (UniqueName: \"kubernetes.io/projected/f195a785-22bc-46c1-833b-f63a37a2de85-kube-api-access-dlkmz\") pod \"maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv\" (UID: \"f195a785-22bc-46c1-833b-f63a37a2de85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv"
Nov 03 08:23:21.325106 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.325053    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv"
Nov 03 08:23:21.676419 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:21.676390    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv"]
Nov 03 08:23:22.306941 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:23:22.306900    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:23:22.486188 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:22.486107    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44"]
Nov 03 08:23:22.496344 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:22.496311    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44"
Nov 03 08:23:22.502461 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:22.502383    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44"]
Nov 03 08:23:22.539502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:22.539470    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/3.log"
Nov 03 08:23:22.543914 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:22.543883    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/3.log"
Nov 03 08:23:22.563110 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:22.562100    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-n28db_5738916f-8202-40ed-823e-90f7c9cc4ec9/csi-driver/0.log"
Nov 03 08:23:22.563731 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:22.563702    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-n28db_5738916f-8202-40ed-823e-90f7c9cc4ec9/csi-driver/0.log"
Nov 03 08:23:22.600944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:22.600369    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq5n6\" (UniqueName: \"kubernetes.io/projected/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151-kube-api-access-hq5n6\") pod \"maas-api-revoke-sub-2426787c-cwn44\" (UID: \"1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44"
Nov 03 08:23:22.600944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:22.600504    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2426787c-cwn44\" (UID: \"1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44"
Nov 03 08:23:22.600944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:22.600650    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2426787c-cwn44\" (UID: \"1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44"
Nov 03 08:23:22.681732 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:22.681683    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv" event={"ID":"f195a785-22bc-46c1-833b-f63a37a2de85","Type":"ContainerStarted","Data":"5e6ca19ee13217660a024a0597350c1d305d864db6499eea820295dd5614da9e"}
Nov 03 08:23:22.681896 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:22.681842    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv" event={"ID":"f195a785-22bc-46c1-833b-f63a37a2de85","Type":"ContainerStarted","Data":"72200ecdd220d088a220adf33eabb82672cbf9d502b0a9467111abbb7d522ac6"}
Nov 03 08:23:22.696882 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:22.696835    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv" podStartSLOduration=2.696819353 podStartE2EDuration="2.696819353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:23:22.695092189 +0000 UTC m=+1800.809556120" watchObservedRunningTime="2026-11-03 08:23:22.696819353 +0000 UTC m=+1800.811283285"
Nov 03 08:23:22.702074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:22.702052    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq5n6\" (UniqueName: \"kubernetes.io/projected/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151-kube-api-access-hq5n6\") pod \"maas-api-revoke-sub-2426787c-cwn44\" (UID: \"1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44"
Nov 03 08:23:22.702194 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:22.702144    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2426787c-cwn44\" (UID: \"1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44"
Nov 03 08:23:22.702269 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:22.702254    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2426787c-cwn44\" (UID: \"1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44"
Nov 03 08:23:22.702815 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:22.702751    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2426787c-cwn44\" (UID: \"1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44"
Nov 03 08:23:22.704985 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:22.704965    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2426787c-cwn44\" (UID: \"1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44"
Nov 03 08:23:22.709489 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:22.709464    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq5n6\" (UniqueName: \"kubernetes.io/projected/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151-kube-api-access-hq5n6\") pod \"maas-api-revoke-sub-2426787c-cwn44\" (UID: \"1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44"
Nov 03 08:23:22.836526 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:22.836497    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44"
Nov 03 08:23:23.118871 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:23:23.118840    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1bfff4_be24_4fdf_a2a8_57cc0f2c9151.slice/crio-54d67fb2dc8e95269b760707923572fe68a757a049e63ed8f4034e16eb9325ce WatchSource:0}: Error finding container 54d67fb2dc8e95269b760707923572fe68a757a049e63ed8f4034e16eb9325ce: Status 404 returned error can't find the container with id 54d67fb2dc8e95269b760707923572fe68a757a049e63ed8f4034e16eb9325ce
Nov 03 08:23:23.120045 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:23.120020    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44"]
Nov 03 08:23:23.675917 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:23.675850    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m"]
Nov 03 08:23:23.690475 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:23.690443    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb"]
Nov 03 08:23:23.695987 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:23.695960    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44" event={"ID":"1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151","Type":"ContainerStarted","Data":"bf121ce95868d2df9d674aa325b9bd9b9f6f03832b90dd857260a3150136a6be"}
Nov 03 08:23:23.696162 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:23.696016    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44" event={"ID":"1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151","Type":"ContainerStarted","Data":"54d67fb2dc8e95269b760707923572fe68a757a049e63ed8f4034e16eb9325ce"}
Nov 03 08:23:23.698188 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:23.698164    5070 generic.go:356] "Generic (PLEG): container finished" podID="f195a785-22bc-46c1-833b-f63a37a2de85" containerID="5e6ca19ee13217660a024a0597350c1d305d864db6499eea820295dd5614da9e" exitCode=0
Nov 03 08:23:23.698317 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:23.698253    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv" event={"ID":"f195a785-22bc-46c1-833b-f63a37a2de85","Type":"ContainerDied","Data":"5e6ca19ee13217660a024a0597350c1d305d864db6499eea820295dd5614da9e"}
Nov 03 08:23:23.727203 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:23.727150    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44" podStartSLOduration=1.7271338790000001 podStartE2EDuration="1.727133879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:23:23.723332241 +0000 UTC m=+1801.837796175" watchObservedRunningTime="2026-11-03 08:23:23.727133879 +0000 UTC m=+1801.841597810"
Nov 03 08:23:24.643528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:24.643490    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh"]
Nov 03 08:23:24.653084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:24.653051    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh"
Nov 03 08:23:24.657344 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:24.657312    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh"]
Nov 03 08:23:24.713221 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:24.713190    5070 generic.go:356] "Generic (PLEG): container finished" podID="1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151" containerID="bf121ce95868d2df9d674aa325b9bd9b9f6f03832b90dd857260a3150136a6be" exitCode=0
Nov 03 08:23:24.713624 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:24.713236    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44" event={"ID":"1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151","Type":"ContainerDied","Data":"bf121ce95868d2df9d674aa325b9bd9b9f6f03832b90dd857260a3150136a6be"}
Nov 03 08:23:24.815784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:24.810839    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edddbc48-8787-4a3e-bdce-3f7740681280-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d710f944-7jvsh\" (UID: \"edddbc48-8787-4a3e-bdce-3f7740681280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh"
Nov 03 08:23:24.815784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:24.810958    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edddbc48-8787-4a3e-bdce-3f7740681280-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d710f944-7jvsh\" (UID: \"edddbc48-8787-4a3e-bdce-3f7740681280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh"
Nov 03 08:23:24.815784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:24.811171    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd4hz\" (UniqueName: \"kubernetes.io/projected/edddbc48-8787-4a3e-bdce-3f7740681280-kube-api-access-dd4hz\") pod \"maas-api-revoke-sub-d710f944-7jvsh\" (UID: \"edddbc48-8787-4a3e-bdce-3f7740681280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh"
Nov 03 08:23:24.912887 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:24.912806    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd4hz\" (UniqueName: \"kubernetes.io/projected/edddbc48-8787-4a3e-bdce-3f7740681280-kube-api-access-dd4hz\") pod \"maas-api-revoke-sub-d710f944-7jvsh\" (UID: \"edddbc48-8787-4a3e-bdce-3f7740681280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh"
Nov 03 08:23:24.913098 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:24.913074    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edddbc48-8787-4a3e-bdce-3f7740681280-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d710f944-7jvsh\" (UID: \"edddbc48-8787-4a3e-bdce-3f7740681280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh"
Nov 03 08:23:24.913200 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:24.913160    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edddbc48-8787-4a3e-bdce-3f7740681280-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d710f944-7jvsh\" (UID: \"edddbc48-8787-4a3e-bdce-3f7740681280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh"
Nov 03 08:23:24.914153 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:24.914128    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edddbc48-8787-4a3e-bdce-3f7740681280-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d710f944-7jvsh\" (UID: \"edddbc48-8787-4a3e-bdce-3f7740681280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh"
Nov 03 08:23:24.917042 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:24.917016    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edddbc48-8787-4a3e-bdce-3f7740681280-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d710f944-7jvsh\" (UID: \"edddbc48-8787-4a3e-bdce-3f7740681280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh"
Nov 03 08:23:24.927014 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:24.926994    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd4hz\" (UniqueName: \"kubernetes.io/projected/edddbc48-8787-4a3e-bdce-3f7740681280-kube-api-access-dd4hz\") pod \"maas-api-revoke-sub-d710f944-7jvsh\" (UID: \"edddbc48-8787-4a3e-bdce-3f7740681280\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh"
Nov 03 08:23:24.980811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:24.980782    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh"
Nov 03 08:23:25.040610 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.040579    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv"
Nov 03 08:23:25.218798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.218690    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f195a785-22bc-46c1-833b-f63a37a2de85-kube-api-access-dlkmz\" (UniqueName: \"kubernetes.io/projected/f195a785-22bc-46c1-833b-f63a37a2de85-kube-api-access-dlkmz\") pod \"f195a785-22bc-46c1-833b-f63a37a2de85\" (UID: \"f195a785-22bc-46c1-833b-f63a37a2de85\") "
Nov 03 08:23:25.218798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.218778    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f195a785-22bc-46c1-833b-f63a37a2de85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f195a785-22bc-46c1-833b-f63a37a2de85-maas-api-xks-ca\") pod \"f195a785-22bc-46c1-833b-f63a37a2de85\" (UID: \"f195a785-22bc-46c1-833b-f63a37a2de85\") "
Nov 03 08:23:25.218983 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.218819    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f195a785-22bc-46c1-833b-f63a37a2de85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f195a785-22bc-46c1-833b-f63a37a2de85-maas-api-openshift-ca\") pod \"f195a785-22bc-46c1-833b-f63a37a2de85\" (UID: \"f195a785-22bc-46c1-833b-f63a37a2de85\") "
Nov 03 08:23:25.219250 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.219224    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f195a785-22bc-46c1-833b-f63a37a2de85-maas-api-openshift-ca" pod "f195a785-22bc-46c1-833b-f63a37a2de85" (UID: "f195a785-22bc-46c1-833b-f63a37a2de85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:25.219321 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.219308    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f195a785-22bc-46c1-833b-f63a37a2de85-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:25.221024 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.220986    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f195a785-22bc-46c1-833b-f63a37a2de85-kube-api-access-dlkmz" pod "f195a785-22bc-46c1-833b-f63a37a2de85" (UID: "f195a785-22bc-46c1-833b-f63a37a2de85"). InnerVolumeSpecName "kube-api-access-dlkmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:25.221024 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.221000    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f195a785-22bc-46c1-833b-f63a37a2de85-maas-api-xks-ca" pod "f195a785-22bc-46c1-833b-f63a37a2de85" (UID: "f195a785-22bc-46c1-833b-f63a37a2de85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:25.273837 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.273808    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh"]
Nov 03 08:23:25.274067 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:23:25.274048    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedddbc48_8787_4a3e_bdce_3f7740681280.slice/crio-08bfc2a16952ba9bfbbbba61e009628542fd001f3ea6a74669a9aef752eee6a3 WatchSource:0}: Error finding container 08bfc2a16952ba9bfbbbba61e009628542fd001f3ea6a74669a9aef752eee6a3: Status 404 returned error can't find the container with id 08bfc2a16952ba9bfbbbba61e009628542fd001f3ea6a74669a9aef752eee6a3
Nov 03 08:23:25.321124 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.321101    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlkmz\" (UniqueName: \"kubernetes.io/projected/f195a785-22bc-46c1-833b-f63a37a2de85-kube-api-access-dlkmz\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:25.321226 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.321130    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f195a785-22bc-46c1-833b-f63a37a2de85-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:25.725141 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.725105    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh" event={"ID":"edddbc48-8787-4a3e-bdce-3f7740681280","Type":"ContainerStarted","Data":"8ea162f343930666669df4c5c4596c179dc694807f3e78b1bbf4fcdd7911ca04"}
Nov 03 08:23:25.725494 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.725207    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh" event={"ID":"edddbc48-8787-4a3e-bdce-3f7740681280","Type":"ContainerStarted","Data":"08bfc2a16952ba9bfbbbba61e009628542fd001f3ea6a74669a9aef752eee6a3"}
Nov 03 08:23:25.726905 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.726884    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv"
Nov 03 08:23:25.726988 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.726917    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv" event={"ID":"f195a785-22bc-46c1-833b-f63a37a2de85","Type":"ContainerDied","Data":"72200ecdd220d088a220adf33eabb82672cbf9d502b0a9467111abbb7d522ac6"}
Nov 03 08:23:25.726988 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.726968    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72200ecdd220d088a220adf33eabb82672cbf9d502b0a9467111abbb7d522ac6"
Nov 03 08:23:25.747175 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.747131    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh" podStartSLOduration=1.747118585 podStartE2EDuration="1.747118585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:23:25.741648427 +0000 UTC m=+1803.856112358" watchObservedRunningTime="2026-11-03 08:23:25.747118585 +0000 UTC m=+1803.861582516"
Nov 03 08:23:25.887704 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.887680    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44"
Nov 03 08:23:25.957838 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.957782    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151-maas-api-xks-ca\") pod \"1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151\" (UID: \"1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151\") "
Nov 03 08:23:25.957997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.957926    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151-maas-api-openshift-ca\") pod \"1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151\" (UID: \"1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151\") "
Nov 03 08:23:25.958293 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.958268    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151-maas-api-openshift-ca" pod "1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151" (UID: "1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:25.958643 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.958628    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:25.960715 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:25.960683    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151-maas-api-xks-ca" pod "1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151" (UID: "1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:26.059643 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:26.059618    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151-kube-api-access-hq5n6\" (UniqueName: \"kubernetes.io/projected/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151-kube-api-access-hq5n6\") pod \"1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151\" (UID: \"1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151\") "
Nov 03 08:23:26.060301 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:26.060282    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:26.061725 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:26.061695    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151-kube-api-access-hq5n6" pod "1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151" (UID: "1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151"). InnerVolumeSpecName "kube-api-access-hq5n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:26.162037 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:26.162014    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq5n6\" (UniqueName: \"kubernetes.io/projected/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151-kube-api-access-hq5n6\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:26.744365 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:26.744331    5070 generic.go:356] "Generic (PLEG): container finished" podID="edddbc48-8787-4a3e-bdce-3f7740681280" containerID="8ea162f343930666669df4c5c4596c179dc694807f3e78b1bbf4fcdd7911ca04" exitCode=0
Nov 03 08:23:26.744920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:26.744728    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh" event={"ID":"edddbc48-8787-4a3e-bdce-3f7740681280","Type":"ContainerDied","Data":"8ea162f343930666669df4c5c4596c179dc694807f3e78b1bbf4fcdd7911ca04"}
Nov 03 08:23:26.746314 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:26.746292    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44" event={"ID":"1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151","Type":"ContainerDied","Data":"54d67fb2dc8e95269b760707923572fe68a757a049e63ed8f4034e16eb9325ce"}
Nov 03 08:23:26.746435 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:26.746317    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44"
Nov 03 08:23:26.746435 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:26.746329    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54d67fb2dc8e95269b760707923572fe68a757a049e63ed8f4034e16eb9325ce"
Nov 03 08:23:27.967107 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:27.967085    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh"
Nov 03 08:23:28.036587 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:28.036517    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edddbc48-8787-4a3e-bdce-3f7740681280-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edddbc48-8787-4a3e-bdce-3f7740681280-maas-api-xks-ca\") pod \"edddbc48-8787-4a3e-bdce-3f7740681280\" (UID: \"edddbc48-8787-4a3e-bdce-3f7740681280\") "
Nov 03 08:23:28.036746 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:28.036642    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edddbc48-8787-4a3e-bdce-3f7740681280-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edddbc48-8787-4a3e-bdce-3f7740681280-maas-api-openshift-ca\") pod \"edddbc48-8787-4a3e-bdce-3f7740681280\" (UID: \"edddbc48-8787-4a3e-bdce-3f7740681280\") "
Nov 03 08:23:28.036840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:28.036803    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edddbc48-8787-4a3e-bdce-3f7740681280-kube-api-access-dd4hz\" (UniqueName: \"kubernetes.io/projected/edddbc48-8787-4a3e-bdce-3f7740681280-kube-api-access-dd4hz\") pod \"edddbc48-8787-4a3e-bdce-3f7740681280\" (UID: \"edddbc48-8787-4a3e-bdce-3f7740681280\") "
Nov 03 08:23:28.037073 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:28.037044    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edddbc48-8787-4a3e-bdce-3f7740681280-maas-api-openshift-ca" pod "edddbc48-8787-4a3e-bdce-3f7740681280" (UID: "edddbc48-8787-4a3e-bdce-3f7740681280"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:28.037864 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:28.037837    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edddbc48-8787-4a3e-bdce-3f7740681280-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:28.039161 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:28.039122    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edddbc48-8787-4a3e-bdce-3f7740681280-kube-api-access-dd4hz" pod "edddbc48-8787-4a3e-bdce-3f7740681280" (UID: "edddbc48-8787-4a3e-bdce-3f7740681280"). InnerVolumeSpecName "kube-api-access-dd4hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:28.040335 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:28.040309    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edddbc48-8787-4a3e-bdce-3f7740681280-maas-api-xks-ca" pod "edddbc48-8787-4a3e-bdce-3f7740681280" (UID: "edddbc48-8787-4a3e-bdce-3f7740681280"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:28.161372 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:28.161133    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edddbc48-8787-4a3e-bdce-3f7740681280-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:28.161372 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:28.161171    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd4hz\" (UniqueName: \"kubernetes.io/projected/edddbc48-8787-4a3e-bdce-3f7740681280-kube-api-access-dd4hz\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:28.676496 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:28.676457    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m" podUID="cb3722fc-983b-4b33-9269-8e559475aff5" containerName="payload-processing" containerID="cri-o://84944f5477498a6baef9b706c16bad14f7b62a33ae6c8ab56da34661d69850c2" gracePeriod=25
Nov 03 08:23:28.690744 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:28.690710    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb" podUID="d449195e-0ed5-4c8e-ac1b-3242198699f7" containerName="payload-pre-processing" containerID="cri-o://5af597feaa1c3de87495549902e7b6a22b74cecb6a7d6392963fd11a76083a22" gracePeriod=25
Nov 03 08:23:28.780720 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:28.780676    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh" event={"ID":"edddbc48-8787-4a3e-bdce-3f7740681280","Type":"ContainerDied","Data":"08bfc2a16952ba9bfbbbba61e009628542fd001f3ea6a74669a9aef752eee6a3"}
Nov 03 08:23:28.780931 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:28.780826    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08bfc2a16952ba9bfbbbba61e009628542fd001f3ea6a74669a9aef752eee6a3"
Nov 03 08:23:28.780931 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:28.780920    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh"
Nov 03 08:23:31.219184 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:31.219144    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn"]
Nov 03 08:23:33.106603 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.106561    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" containerID="cri-o://3858e4e8bb39a20b059d2b818825713fc649efe9b8710271688eca56b6ae2aa3" gracePeriod=45
Nov 03 08:23:33.112240 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.112202    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" containerID="cri-o://d387b38ddd3da9e703275063b08459ba2a68b950ab10f4b9d53dd75cc4752003" gracePeriod=45
Nov 03 08:23:33.227857 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.227828    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/ready\": dial tcp 10.132.0.72:8000: connect: connection refused"
Nov 03 08:23:33.527925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.527903    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:23:33.572616 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.572587    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:23:33.643158 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.643128    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-tmp-dir\") pod \"54c00158-d927-478c-8fee-13f94bbdb2d6\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") "
Nov 03 08:23:33.643352 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.643182    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54c00158-d927-478c-8fee-13f94bbdb2d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/54c00158-d927-478c-8fee-13f94bbdb2d6-tls-certs\") pod \"54c00158-d927-478c-8fee-13f94bbdb2d6\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") "
Nov 03 08:23:33.643352 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.643209    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c00158-d927-478c-8fee-13f94bbdb2d6-kube-api-access-5rz6w\" (UniqueName: \"kubernetes.io/projected/54c00158-d927-478c-8fee-13f94bbdb2d6-kube-api-access-5rz6w\") pod \"54c00158-d927-478c-8fee-13f94bbdb2d6\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") "
Nov 03 08:23:33.643489 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.643378    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-model-cache\") pod \"54c00158-d927-478c-8fee-13f94bbdb2d6\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") "
Nov 03 08:23:33.643586 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.643570    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-dshm\") pod \"54c00158-d927-478c-8fee-13f94bbdb2d6\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") "
Nov 03 08:23:33.643652 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.643604    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-home\" (UniqueName: \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-home\") pod \"54c00158-d927-478c-8fee-13f94bbdb2d6\" (UID: \"54c00158-d927-478c-8fee-13f94bbdb2d6\") "
Nov 03 08:23:33.649313 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.649284    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-home" pod "54c00158-d927-478c-8fee-13f94bbdb2d6" (UID: "54c00158-d927-478c-8fee-13f94bbdb2d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:33.649847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.649688    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-model-cache" pod "54c00158-d927-478c-8fee-13f94bbdb2d6" (UID: "54c00158-d927-478c-8fee-13f94bbdb2d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:33.649847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.649728    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c00158-d927-478c-8fee-13f94bbdb2d6-kube-api-access-5rz6w" pod "54c00158-d927-478c-8fee-13f94bbdb2d6" (UID: "54c00158-d927-478c-8fee-13f94bbdb2d6"). InnerVolumeSpecName "kube-api-access-5rz6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:33.650029 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.649983    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-tmp-dir" pod "54c00158-d927-478c-8fee-13f94bbdb2d6" (UID: "54c00158-d927-478c-8fee-13f94bbdb2d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:33.651440 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.651419    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-dshm" pod "54c00158-d927-478c-8fee-13f94bbdb2d6" (UID: "54c00158-d927-478c-8fee-13f94bbdb2d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:33.653263 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.653217    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54c00158-d927-478c-8fee-13f94bbdb2d6-tls-certs" pod "54c00158-d927-478c-8fee-13f94bbdb2d6" (UID: "54c00158-d927-478c-8fee-13f94bbdb2d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:33.745950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.745873    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-dshm\") pod \"a5fed520-3264-4574-bafa-ac12dde22a02\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") "
Nov 03 08:23:33.745950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.745938    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5fed520-3264-4574-bafa-ac12dde22a02-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fed520-3264-4574-bafa-ac12dde22a02-tls-certs\") pod \"a5fed520-3264-4574-bafa-ac12dde22a02\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") "
Nov 03 08:23:33.746126 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.745993    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5fed520-3264-4574-bafa-ac12dde22a02-kube-api-access-8gzqq\" (UniqueName: \"kubernetes.io/projected/a5fed520-3264-4574-bafa-ac12dde22a02-kube-api-access-8gzqq\") pod \"a5fed520-3264-4574-bafa-ac12dde22a02\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") "
Nov 03 08:23:33.746202 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.746177    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-model-cache\") pod \"a5fed520-3264-4574-bafa-ac12dde22a02\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") "
Nov 03 08:23:33.746457 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.746430    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-home\" (UniqueName: \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-home\") pod \"a5fed520-3264-4574-bafa-ac12dde22a02\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") "
Nov 03 08:23:33.746566 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.746452    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-model-cache" pod "a5fed520-3264-4574-bafa-ac12dde22a02" (UID: "a5fed520-3264-4574-bafa-ac12dde22a02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:33.746566 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.746535    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-tmp-dir\") pod \"a5fed520-3264-4574-bafa-ac12dde22a02\" (UID: \"a5fed520-3264-4574-bafa-ac12dde22a02\") "
Nov 03 08:23:33.746686 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.746625    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-home" pod "a5fed520-3264-4574-bafa-ac12dde22a02" (UID: "a5fed520-3264-4574-bafa-ac12dde22a02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:33.746744 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.746709    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-tmp-dir" pod "a5fed520-3264-4574-bafa-ac12dde22a02" (UID: "a5fed520-3264-4574-bafa-ac12dde22a02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:33.747367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.747337    5070 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-model-cache\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:33.747367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.747367    5070 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-dshm\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:33.747542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.747376    5070 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-home\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:33.747542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.747385    5070 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-home\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:33.747542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.747392    5070 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-tmp-dir\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:33.747542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.747400    5070 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-tmp-dir\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:33.747542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.747408    5070 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54c00158-d927-478c-8fee-13f94bbdb2d6-tls-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:33.747542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.747418    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rz6w\" (UniqueName: \"kubernetes.io/projected/54c00158-d927-478c-8fee-13f94bbdb2d6-kube-api-access-5rz6w\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:33.747542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.747428    5070 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54c00158-d927-478c-8fee-13f94bbdb2d6-model-cache\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:33.748736 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.748713    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-dshm" pod "a5fed520-3264-4574-bafa-ac12dde22a02" (UID: "a5fed520-3264-4574-bafa-ac12dde22a02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:23:33.749158 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.749131    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5fed520-3264-4574-bafa-ac12dde22a02-tls-certs" pod "a5fed520-3264-4574-bafa-ac12dde22a02" (UID: "a5fed520-3264-4574-bafa-ac12dde22a02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:33.749260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.749191    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5fed520-3264-4574-bafa-ac12dde22a02-kube-api-access-8gzqq" pod "a5fed520-3264-4574-bafa-ac12dde22a02" (UID: "a5fed520-3264-4574-bafa-ac12dde22a02"). InnerVolumeSpecName "kube-api-access-8gzqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:33.849116 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.849087    5070 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5fed520-3264-4574-bafa-ac12dde22a02-dshm\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:33.849116 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.849111    5070 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5fed520-3264-4574-bafa-ac12dde22a02-tls-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:33.849266 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.849121    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gzqq\" (UniqueName: \"kubernetes.io/projected/a5fed520-3264-4574-bafa-ac12dde22a02-kube-api-access-8gzqq\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:33.978839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.978809    5070 generic.go:356] "Generic (PLEG): container finished" podID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerID="3858e4e8bb39a20b059d2b818825713fc649efe9b8710271688eca56b6ae2aa3" exitCode=0
Nov 03 08:23:33.978997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.978886    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" event={"ID":"54c00158-d927-478c-8fee-13f94bbdb2d6","Type":"ContainerDied","Data":"3858e4e8bb39a20b059d2b818825713fc649efe9b8710271688eca56b6ae2aa3"}
Nov 03 08:23:33.978997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.978913    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"
Nov 03 08:23:33.978997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.978954    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz" event={"ID":"54c00158-d927-478c-8fee-13f94bbdb2d6","Type":"ContainerDied","Data":"dcbd6ff05692bf9148a384617563bfbbe87d9d7f0a1bb4aa22d44931b539f973"}
Nov 03 08:23:33.978997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.978979    5070 scope.go:122] "RemoveContainer" containerID="3858e4e8bb39a20b059d2b818825713fc649efe9b8710271688eca56b6ae2aa3"
Nov 03 08:23:33.980425 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.980404    5070 generic.go:356] "Generic (PLEG): container finished" podID="a5fed520-3264-4574-bafa-ac12dde22a02" containerID="d387b38ddd3da9e703275063b08459ba2a68b950ab10f4b9d53dd75cc4752003" exitCode=0
Nov 03 08:23:33.980519 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.980445    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"
Nov 03 08:23:33.980519 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.980494    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" event={"ID":"a5fed520-3264-4574-bafa-ac12dde22a02","Type":"ContainerDied","Data":"d387b38ddd3da9e703275063b08459ba2a68b950ab10f4b9d53dd75cc4752003"}
Nov 03 08:23:33.980606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.980533    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" event={"ID":"a5fed520-3264-4574-bafa-ac12dde22a02","Type":"ContainerDied","Data":"01132aebca33318c8af6200e6fcf153c77dd48852bfb0ae10a08ae6f997def21"}
Nov 03 08:23:33.996214 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.996191    5070 scope.go:122] "RemoveContainer" containerID="3858e4e8bb39a20b059d2b818825713fc649efe9b8710271688eca56b6ae2aa3"
Nov 03 08:23:33.996526 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:23:33.996501    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3858e4e8bb39a20b059d2b818825713fc649efe9b8710271688eca56b6ae2aa3\": container with ID starting with 3858e4e8bb39a20b059d2b818825713fc649efe9b8710271688eca56b6ae2aa3 not found: ID does not exist" containerID="3858e4e8bb39a20b059d2b818825713fc649efe9b8710271688eca56b6ae2aa3"
Nov 03 08:23:33.996658 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.996536    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3858e4e8bb39a20b059d2b818825713fc649efe9b8710271688eca56b6ae2aa3"} err="failed to get container status \"3858e4e8bb39a20b059d2b818825713fc649efe9b8710271688eca56b6ae2aa3\": rpc error: code = NotFound desc = could not find container \"3858e4e8bb39a20b059d2b818825713fc649efe9b8710271688eca56b6ae2aa3\": container with ID starting with 3858e4e8bb39a20b059d2b818825713fc649efe9b8710271688eca56b6ae2aa3 not found: ID does not exist"
Nov 03 08:23:33.996658 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:33.996556    5070 scope.go:122] "RemoveContainer" containerID="d387b38ddd3da9e703275063b08459ba2a68b950ab10f4b9d53dd75cc4752003"
Nov 03 08:23:34.009527 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:34.009509    5070 scope.go:122] "RemoveContainer" containerID="d387b38ddd3da9e703275063b08459ba2a68b950ab10f4b9d53dd75cc4752003"
Nov 03 08:23:34.009847 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:23:34.009828    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d387b38ddd3da9e703275063b08459ba2a68b950ab10f4b9d53dd75cc4752003\": container with ID starting with d387b38ddd3da9e703275063b08459ba2a68b950ab10f4b9d53dd75cc4752003 not found: ID does not exist" containerID="d387b38ddd3da9e703275063b08459ba2a68b950ab10f4b9d53dd75cc4752003"
Nov 03 08:23:34.009933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:34.009854    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d387b38ddd3da9e703275063b08459ba2a68b950ab10f4b9d53dd75cc4752003"} err="failed to get container status \"d387b38ddd3da9e703275063b08459ba2a68b950ab10f4b9d53dd75cc4752003\": rpc error: code = NotFound desc = could not find container \"d387b38ddd3da9e703275063b08459ba2a68b950ab10f4b9d53dd75cc4752003\": container with ID starting with d387b38ddd3da9e703275063b08459ba2a68b950ab10f4b9d53dd75cc4752003 not found: ID does not exist"
Nov 03 08:23:34.010414 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:34.010398    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"]
Nov 03 08:23:34.014978 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:34.014958    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7"]
Nov 03 08:23:34.029621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:34.029596    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" path="/var/lib/kubelet/pods/a5fed520-3264-4574-bafa-ac12dde22a02/volumes"
Nov 03 08:23:34.029966 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:34.029953    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"]
Nov 03 08:23:34.030018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:34.029979    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-15a633/simulator-w4-15a633-kserve-55b977b85d-lgpmz"]
Nov 03 08:23:34.429967 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:34.429649    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-15a633/premium-w4-15a633-kserve-75575b7cf5-qzmk7" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded"
Nov 03 08:23:36.028184 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:36.028150    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" path="/var/lib/kubelet/pods/54c00158-d927-478c-8fee-13f94bbdb2d6/volumes"
Nov 03 08:23:36.229056 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:36.225956    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn" podUID="063e3656-6675-4336-a3c1-cf96b568be28" containerName="payload-processing" containerID="cri-o://89d6a74919c8b739e8ca254ff8e9552e65775ceab1360487a2b6a05bb8333217" gracePeriod=25
Nov 03 08:23:37.193682 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.193647    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts"]
Nov 03 08:23:37.195495 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.195466    5070 cpu_manager.go:414] "containerMap: removing container" podUID="1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151" containerName="revoke-keys"
Nov 03 08:23:37.195710 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.195695    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151" containerName="revoke-keys"
Nov 03 08:23:37.195854 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.195839    5070 cpu_manager.go:414] "containerMap: removing container" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main"
Nov 03 08:23:37.196430 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.195954    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main"
Nov 03 08:23:37.196430 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.195974    5070 cpu_manager.go:414] "containerMap: removing container" podUID="edddbc48-8787-4a3e-bdce-3f7740681280" containerName="revoke-keys"
Nov 03 08:23:37.196430 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.195982    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edddbc48-8787-4a3e-bdce-3f7740681280" containerName="revoke-keys"
Nov 03 08:23:37.196430 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.196194    5070 cpu_manager.go:414] "containerMap: removing container" podUID="f195a785-22bc-46c1-833b-f63a37a2de85" containerName="revoke-keys"
Nov 03 08:23:37.196430 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.196208    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f195a785-22bc-46c1-833b-f63a37a2de85" containerName="revoke-keys"
Nov 03 08:23:37.196430 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.196225    5070 cpu_manager.go:414] "containerMap: removing container" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main"
Nov 03 08:23:37.196430 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.196232    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main"
Nov 03 08:23:37.197135 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.197115    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="f195a785-22bc-46c1-833b-f63a37a2de85" containerName="revoke-keys"
Nov 03 08:23:37.197245 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.197234    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5fed520-3264-4574-bafa-ac12dde22a02" containerName="main"
Nov 03 08:23:37.197456 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.197441    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151" containerName="revoke-keys"
Nov 03 08:23:37.197572 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.197561    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c00158-d927-478c-8fee-13f94bbdb2d6" containerName="main"
Nov 03 08:23:37.197661 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.197651    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="edddbc48-8787-4a3e-bdce-3f7740681280" containerName="revoke-keys"
Nov 03 08:23:37.216438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.216408    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts"
Nov 03 08:23:37.224879 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.224847    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts"]
Nov 03 08:23:37.238785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.226053    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:23:37.238785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.226208    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:23:37.238785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.226291    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:23:37.238785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.226338    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:23:37.382676 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.382635    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/920822a5-777d-42f1-af1f-d0cff4eb1e52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts\" (UID: \"920822a5-777d-42f1-af1f-d0cff4eb1e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts"
Nov 03 08:23:37.382962 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.382935    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/920822a5-777d-42f1-af1f-d0cff4eb1e52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts\" (UID: \"920822a5-777d-42f1-af1f-d0cff4eb1e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts"
Nov 03 08:23:37.383272 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.383249    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpxrg\" (UniqueName: \"kubernetes.io/projected/920822a5-777d-42f1-af1f-d0cff4eb1e52-kube-api-access-xpxrg\") pod \"maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts\" (UID: \"920822a5-777d-42f1-af1f-d0cff4eb1e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts"
Nov 03 08:23:37.485403 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.485321    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/920822a5-777d-42f1-af1f-d0cff4eb1e52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts\" (UID: \"920822a5-777d-42f1-af1f-d0cff4eb1e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts"
Nov 03 08:23:37.485403 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.485357    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/920822a5-777d-42f1-af1f-d0cff4eb1e52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts\" (UID: \"920822a5-777d-42f1-af1f-d0cff4eb1e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts"
Nov 03 08:23:37.485606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.485452    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpxrg\" (UniqueName: \"kubernetes.io/projected/920822a5-777d-42f1-af1f-d0cff4eb1e52-kube-api-access-xpxrg\") pod \"maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts\" (UID: \"920822a5-777d-42f1-af1f-d0cff4eb1e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts"
Nov 03 08:23:37.486148 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.486126    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/920822a5-777d-42f1-af1f-d0cff4eb1e52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts\" (UID: \"920822a5-777d-42f1-af1f-d0cff4eb1e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts"
Nov 03 08:23:37.487940 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.487914    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/920822a5-777d-42f1-af1f-d0cff4eb1e52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts\" (UID: \"920822a5-777d-42f1-af1f-d0cff4eb1e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts"
Nov 03 08:23:37.492634 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.492611    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpxrg\" (UniqueName: \"kubernetes.io/projected/920822a5-777d-42f1-af1f-d0cff4eb1e52-kube-api-access-xpxrg\") pod \"maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts\" (UID: \"920822a5-777d-42f1-af1f-d0cff4eb1e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts"
Nov 03 08:23:37.691604 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:37.691574    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts"
Nov 03 08:23:38.068273 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.068239    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts"]
Nov 03 08:23:38.072309 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.072279    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts" event={"ID":"920822a5-777d-42f1-af1f-d0cff4eb1e52","Type":"ContainerStarted","Data":"c0cfb6e9ae1d63ad125fcac20f8bf74efbe4eb219798d2cd0ab815d277d89f13"}
Nov 03 08:23:38.216175 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.216127    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn" podUID="063e3656-6675-4336-a3c1-cf96b568be28" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:23:38.217711 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.217677    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb" podUID="d449195e-0ed5-4c8e-ac1b-3242198699f7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:23:38.218194 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.218169    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m" podUID="cb3722fc-983b-4b33-9269-8e559475aff5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:23:38.541292 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.540711    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg"]
Nov 03 08:23:38.569728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.560997    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg"
Nov 03 08:23:38.579194 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.579170    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg"]
Nov 03 08:23:38.650854 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.650818    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgpxw\" (UniqueName: \"kubernetes.io/projected/afa12a2a-bfc8-46f0-9647-1ee6f00447d4-kube-api-access-dgpxw\") pod \"maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg\" (UID: \"afa12a2a-bfc8-46f0-9647-1ee6f00447d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg"
Nov 03 08:23:38.650987 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.650894    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa12a2a-bfc8-46f0-9647-1ee6f00447d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg\" (UID: \"afa12a2a-bfc8-46f0-9647-1ee6f00447d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg"
Nov 03 08:23:38.651175 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.651150    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa12a2a-bfc8-46f0-9647-1ee6f00447d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg\" (UID: \"afa12a2a-bfc8-46f0-9647-1ee6f00447d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg"
Nov 03 08:23:38.752480 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.752439    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa12a2a-bfc8-46f0-9647-1ee6f00447d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg\" (UID: \"afa12a2a-bfc8-46f0-9647-1ee6f00447d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg"
Nov 03 08:23:38.754180 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.753359    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa12a2a-bfc8-46f0-9647-1ee6f00447d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg\" (UID: \"afa12a2a-bfc8-46f0-9647-1ee6f00447d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg"
Nov 03 08:23:38.754591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.754568    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgpxw\" (UniqueName: \"kubernetes.io/projected/afa12a2a-bfc8-46f0-9647-1ee6f00447d4-kube-api-access-dgpxw\") pod \"maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg\" (UID: \"afa12a2a-bfc8-46f0-9647-1ee6f00447d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg"
Nov 03 08:23:38.755024 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.755002    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa12a2a-bfc8-46f0-9647-1ee6f00447d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg\" (UID: \"afa12a2a-bfc8-46f0-9647-1ee6f00447d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg"
Nov 03 08:23:38.757997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.757970    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa12a2a-bfc8-46f0-9647-1ee6f00447d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg\" (UID: \"afa12a2a-bfc8-46f0-9647-1ee6f00447d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg"
Nov 03 08:23:38.761693 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.761670    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgpxw\" (UniqueName: \"kubernetes.io/projected/afa12a2a-bfc8-46f0-9647-1ee6f00447d4-kube-api-access-dgpxw\") pod \"maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg\" (UID: \"afa12a2a-bfc8-46f0-9647-1ee6f00447d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg"
Nov 03 08:23:38.887714 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.887682    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg"
Nov 03 08:23:38.915823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.915792    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046740e4-g9hl5"]
Nov 03 08:23:38.924158 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.923036    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046740e4-g9hl5"
Nov 03 08:23:38.927632 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.927602    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046740e4-g9hl5"]
Nov 03 08:23:38.940877 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.940842    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq"]
Nov 03 08:23:38.958698 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.958663    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d8ac2c-99c2-4125-ae50-cd90c6a78609-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046740e4-g9hl5\" (UID: \"20d8ac2c-99c2-4125-ae50-cd90c6a78609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046740e4-g9hl5"
Nov 03 08:23:38.958901 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.958715    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bhgp\" (UniqueName: \"kubernetes.io/projected/20d8ac2c-99c2-4125-ae50-cd90c6a78609-kube-api-access-5bhgp\") pod \"maas-api-revoke-sub-046740e4-g9hl5\" (UID: \"20d8ac2c-99c2-4125-ae50-cd90c6a78609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046740e4-g9hl5"
Nov 03 08:23:38.959067 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.959041    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d8ac2c-99c2-4125-ae50-cd90c6a78609-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046740e4-g9hl5\" (UID: \"20d8ac2c-99c2-4125-ae50-cd90c6a78609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046740e4-g9hl5"
Nov 03 08:23:38.960697 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.960592    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq"]
Nov 03 08:23:38.960837 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:38.960714    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq"
Nov 03 08:23:39.060645 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.060607    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d8ac2c-99c2-4125-ae50-cd90c6a78609-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046740e4-g9hl5\" (UID: \"20d8ac2c-99c2-4125-ae50-cd90c6a78609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046740e4-g9hl5"
Nov 03 08:23:39.060873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.060659    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bhgp\" (UniqueName: \"kubernetes.io/projected/20d8ac2c-99c2-4125-ae50-cd90c6a78609-kube-api-access-5bhgp\") pod \"maas-api-revoke-sub-046740e4-g9hl5\" (UID: \"20d8ac2c-99c2-4125-ae50-cd90c6a78609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046740e4-g9hl5"
Nov 03 08:23:39.060873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.060696    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-886zs\" (UniqueName: \"kubernetes.io/projected/4250e197-3658-4c5d-8130-35878b72827d-kube-api-access-886zs\") pod \"maas-api-revoke-sub-ef94017c-zz5fq\" (UID: \"4250e197-3658-4c5d-8130-35878b72827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq"
Nov 03 08:23:39.060873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.060728    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4250e197-3658-4c5d-8130-35878b72827d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef94017c-zz5fq\" (UID: \"4250e197-3658-4c5d-8130-35878b72827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq"
Nov 03 08:23:39.060873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.060819    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4250e197-3658-4c5d-8130-35878b72827d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef94017c-zz5fq\" (UID: \"4250e197-3658-4c5d-8130-35878b72827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq"
Nov 03 08:23:39.061113 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.060954    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d8ac2c-99c2-4125-ae50-cd90c6a78609-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046740e4-g9hl5\" (UID: \"20d8ac2c-99c2-4125-ae50-cd90c6a78609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046740e4-g9hl5"
Nov 03 08:23:39.061993 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.061959    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d8ac2c-99c2-4125-ae50-cd90c6a78609-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046740e4-g9hl5\" (UID: \"20d8ac2c-99c2-4125-ae50-cd90c6a78609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046740e4-g9hl5"
Nov 03 08:23:39.063134 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.063110    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d8ac2c-99c2-4125-ae50-cd90c6a78609-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046740e4-g9hl5\" (UID: \"20d8ac2c-99c2-4125-ae50-cd90c6a78609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046740e4-g9hl5"
Nov 03 08:23:39.067937 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.067912    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bhgp\" (UniqueName: \"kubernetes.io/projected/20d8ac2c-99c2-4125-ae50-cd90c6a78609-kube-api-access-5bhgp\") pod \"maas-api-revoke-sub-046740e4-g9hl5\" (UID: \"20d8ac2c-99c2-4125-ae50-cd90c6a78609\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046740e4-g9hl5"
Nov 03 08:23:39.085898 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.085869    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts" event={"ID":"920822a5-777d-42f1-af1f-d0cff4eb1e52","Type":"ContainerStarted","Data":"4a1999b3bfc18a679661b7a31750322a8efd2dcb638b078ee570a69b745ea8da"}
Nov 03 08:23:39.103486 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.103450    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts" podStartSLOduration=2.103439072 podStartE2EDuration="2.103439072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:23:39.099148008 +0000 UTC m=+1817.213611938" watchObservedRunningTime="2026-11-03 08:23:39.103439072 +0000 UTC m=+1817.217903034"
Nov 03 08:23:39.162323 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.162257    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-886zs\" (UniqueName: \"kubernetes.io/projected/4250e197-3658-4c5d-8130-35878b72827d-kube-api-access-886zs\") pod \"maas-api-revoke-sub-ef94017c-zz5fq\" (UID: \"4250e197-3658-4c5d-8130-35878b72827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq"
Nov 03 08:23:39.162323 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.162298    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4250e197-3658-4c5d-8130-35878b72827d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef94017c-zz5fq\" (UID: \"4250e197-3658-4c5d-8130-35878b72827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq"
Nov 03 08:23:39.162501 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.162368    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4250e197-3658-4c5d-8130-35878b72827d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef94017c-zz5fq\" (UID: \"4250e197-3658-4c5d-8130-35878b72827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq"
Nov 03 08:23:39.163040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.163019    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4250e197-3658-4c5d-8130-35878b72827d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef94017c-zz5fq\" (UID: \"4250e197-3658-4c5d-8130-35878b72827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq"
Nov 03 08:23:39.164725 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.164705    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4250e197-3658-4c5d-8130-35878b72827d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef94017c-zz5fq\" (UID: \"4250e197-3658-4c5d-8130-35878b72827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq"
Nov 03 08:23:39.169924 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.169903    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-886zs\" (UniqueName: \"kubernetes.io/projected/4250e197-3658-4c5d-8130-35878b72827d-kube-api-access-886zs\") pod \"maas-api-revoke-sub-ef94017c-zz5fq\" (UID: \"4250e197-3658-4c5d-8130-35878b72827d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq"
Nov 03 08:23:39.189531 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:23:39.189510    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa12a2a_bfc8_46f0_9647_1ee6f00447d4.slice/crio-d925beace2811dfa5718f044e5752bfea2ecdef76d092941ccef553029bd0cef WatchSource:0}: Error finding container d925beace2811dfa5718f044e5752bfea2ecdef76d092941ccef553029bd0cef: Status 404 returned error can't find the container with id d925beace2811dfa5718f044e5752bfea2ecdef76d092941ccef553029bd0cef
Nov 03 08:23:39.190267 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.190246    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg"]
Nov 03 08:23:39.269274 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.269253    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046740e4-g9hl5"
Nov 03 08:23:39.286548 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.286524    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq"
Nov 03 08:23:39.635236 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:23:39.635202    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d8ac2c_99c2_4125_ae50_cd90c6a78609.slice/crio-3f2bf9ad7cd48f4b64cef5d0c556df665a4f438df9c96c7b59386a0706ab3902 WatchSource:0}: Error finding container 3f2bf9ad7cd48f4b64cef5d0c556df665a4f438df9c96c7b59386a0706ab3902: Status 404 returned error can't find the container with id 3f2bf9ad7cd48f4b64cef5d0c556df665a4f438df9c96c7b59386a0706ab3902
Nov 03 08:23:39.635969 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.635869    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046740e4-g9hl5"]
Nov 03 08:23:39.702201 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:39.702177    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq"]
Nov 03 08:23:39.702865 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:23:39.702833    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4250e197_3658_4c5d_8130_35878b72827d.slice/crio-94efc8a697764739198d6bd16cb950e48acd382a71cf971efb97e11968510ade WatchSource:0}: Error finding container 94efc8a697764739198d6bd16cb950e48acd382a71cf971efb97e11968510ade: Status 404 returned error can't find the container with id 94efc8a697764739198d6bd16cb950e48acd382a71cf971efb97e11968510ade
Nov 03 08:23:40.096123 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:40.096088    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046740e4-g9hl5" event={"ID":"20d8ac2c-99c2-4125-ae50-cd90c6a78609","Type":"ContainerStarted","Data":"3f2bf9ad7cd48f4b64cef5d0c556df665a4f438df9c96c7b59386a0706ab3902"}
Nov 03 08:23:40.097900 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:40.097871    5070 generic.go:356] "Generic (PLEG): container finished" podID="920822a5-777d-42f1-af1f-d0cff4eb1e52" containerID="4a1999b3bfc18a679661b7a31750322a8efd2dcb638b078ee570a69b745ea8da" exitCode=0
Nov 03 08:23:40.098044 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:40.097998    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts" event={"ID":"920822a5-777d-42f1-af1f-d0cff4eb1e52","Type":"ContainerDied","Data":"4a1999b3bfc18a679661b7a31750322a8efd2dcb638b078ee570a69b745ea8da"}
Nov 03 08:23:40.099956 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:40.099928    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq" event={"ID":"4250e197-3658-4c5d-8130-35878b72827d","Type":"ContainerStarted","Data":"270f5e4064f4a48f7a1821756bdc6d0c984e03c9f80543bb13a67148f9b2970a"}
Nov 03 08:23:40.100092 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:40.099980    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq" event={"ID":"4250e197-3658-4c5d-8130-35878b72827d","Type":"ContainerStarted","Data":"94efc8a697764739198d6bd16cb950e48acd382a71cf971efb97e11968510ade"}
Nov 03 08:23:40.101698 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:40.101674    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg" event={"ID":"afa12a2a-bfc8-46f0-9647-1ee6f00447d4","Type":"ContainerStarted","Data":"82255fdd3cdedf7f896fc6322dfe16b6b3df9fd8e66876be1564e2cd769ac487"}
Nov 03 08:23:40.101813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:40.101712    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg" event={"ID":"afa12a2a-bfc8-46f0-9647-1ee6f00447d4","Type":"ContainerStarted","Data":"d925beace2811dfa5718f044e5752bfea2ecdef76d092941ccef553029bd0cef"}
Nov 03 08:23:40.127593 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:40.127546    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq" podStartSLOduration=2.127534328 podStartE2EDuration="2.127534328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:23:40.126060679 +0000 UTC m=+1818.240524611" watchObservedRunningTime="2026-11-03 08:23:40.127534328 +0000 UTC m=+1818.241998259"
Nov 03 08:23:40.144576 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:40.144538    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg" podStartSLOduration=2.144526147 podStartE2EDuration="2.144526147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:23:40.139277382 +0000 UTC m=+1818.253741315" watchObservedRunningTime="2026-11-03 08:23:40.144526147 +0000 UTC m=+1818.258990072"
Nov 03 08:23:41.111217 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:41.111184    5070 generic.go:356] "Generic (PLEG): container finished" podID="4250e197-3658-4c5d-8130-35878b72827d" containerID="270f5e4064f4a48f7a1821756bdc6d0c984e03c9f80543bb13a67148f9b2970a" exitCode=0
Nov 03 08:23:41.111630 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:41.111272    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq" event={"ID":"4250e197-3658-4c5d-8130-35878b72827d","Type":"ContainerDied","Data":"270f5e4064f4a48f7a1821756bdc6d0c984e03c9f80543bb13a67148f9b2970a"}
Nov 03 08:23:41.112944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:41.112922    5070 generic.go:356] "Generic (PLEG): container finished" podID="afa12a2a-bfc8-46f0-9647-1ee6f00447d4" containerID="82255fdd3cdedf7f896fc6322dfe16b6b3df9fd8e66876be1564e2cd769ac487" exitCode=0
Nov 03 08:23:41.113070 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:41.113006    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg" event={"ID":"afa12a2a-bfc8-46f0-9647-1ee6f00447d4","Type":"ContainerDied","Data":"82255fdd3cdedf7f896fc6322dfe16b6b3df9fd8e66876be1564e2cd769ac487"}
Nov 03 08:23:41.114594 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:41.114572    5070 generic.go:356] "Generic (PLEG): container finished" podID="20d8ac2c-99c2-4125-ae50-cd90c6a78609" containerID="d3021274e97b01f31e9a7d9d62be1b1b7649c569433a8fdbed24423511cf841a" exitCode=0
Nov 03 08:23:41.114708 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:41.114629    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046740e4-g9hl5" event={"ID":"20d8ac2c-99c2-4125-ae50-cd90c6a78609","Type":"ContainerDied","Data":"d3021274e97b01f31e9a7d9d62be1b1b7649c569433a8fdbed24423511cf841a"}
Nov 03 08:23:41.360012 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:41.359978    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts"
Nov 03 08:23:41.455591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:41.455499    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/920822a5-777d-42f1-af1f-d0cff4eb1e52-kube-api-access-xpxrg\" (UniqueName: \"kubernetes.io/projected/920822a5-777d-42f1-af1f-d0cff4eb1e52-kube-api-access-xpxrg\") pod \"920822a5-777d-42f1-af1f-d0cff4eb1e52\" (UID: \"920822a5-777d-42f1-af1f-d0cff4eb1e52\") "
Nov 03 08:23:41.456169 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:41.455686    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/920822a5-777d-42f1-af1f-d0cff4eb1e52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/920822a5-777d-42f1-af1f-d0cff4eb1e52-maas-api-openshift-ca\") pod \"920822a5-777d-42f1-af1f-d0cff4eb1e52\" (UID: \"920822a5-777d-42f1-af1f-d0cff4eb1e52\") "
Nov 03 08:23:41.456169 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:41.455740    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/920822a5-777d-42f1-af1f-d0cff4eb1e52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/920822a5-777d-42f1-af1f-d0cff4eb1e52-maas-api-xks-ca\") pod \"920822a5-777d-42f1-af1f-d0cff4eb1e52\" (UID: \"920822a5-777d-42f1-af1f-d0cff4eb1e52\") "
Nov 03 08:23:41.457291 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:41.457264    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920822a5-777d-42f1-af1f-d0cff4eb1e52-maas-api-openshift-ca" pod "920822a5-777d-42f1-af1f-d0cff4eb1e52" (UID: "920822a5-777d-42f1-af1f-d0cff4eb1e52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:41.459261 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:41.459231    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/920822a5-777d-42f1-af1f-d0cff4eb1e52-maas-api-xks-ca" pod "920822a5-777d-42f1-af1f-d0cff4eb1e52" (UID: "920822a5-777d-42f1-af1f-d0cff4eb1e52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:41.459800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:41.459736    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920822a5-777d-42f1-af1f-d0cff4eb1e52-kube-api-access-xpxrg" pod "920822a5-777d-42f1-af1f-d0cff4eb1e52" (UID: "920822a5-777d-42f1-af1f-d0cff4eb1e52"). InnerVolumeSpecName "kube-api-access-xpxrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:41.564990 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:41.560075    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpxrg\" (UniqueName: \"kubernetes.io/projected/920822a5-777d-42f1-af1f-d0cff4eb1e52-kube-api-access-xpxrg\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:41.564990 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:41.560111    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/920822a5-777d-42f1-af1f-d0cff4eb1e52-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:41.564990 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:41.560127    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/920822a5-777d-42f1-af1f-d0cff4eb1e52-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:42.207791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:42.207681    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts"
Nov 03 08:23:42.214780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:42.208462    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts" event={"ID":"920822a5-777d-42f1-af1f-d0cff4eb1e52","Type":"ContainerDied","Data":"c0cfb6e9ae1d63ad125fcac20f8bf74efbe4eb219798d2cd0ab815d277d89f13"}
Nov 03 08:23:42.214780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:42.208508    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0cfb6e9ae1d63ad125fcac20f8bf74efbe4eb219798d2cd0ab815d277d89f13"
Nov 03 08:23:43.130235 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.130208    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046740e4-g9hl5"
Nov 03 08:23:43.133937 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.133917    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq"
Nov 03 08:23:43.137817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.137797    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg"
Nov 03 08:23:43.220294 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.220225    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq"
Nov 03 08:23:43.220294 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.220257    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq" event={"ID":"4250e197-3658-4c5d-8130-35878b72827d","Type":"ContainerDied","Data":"94efc8a697764739198d6bd16cb950e48acd382a71cf971efb97e11968510ade"}
Nov 03 08:23:43.220721 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.220304    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94efc8a697764739198d6bd16cb950e48acd382a71cf971efb97e11968510ade"
Nov 03 08:23:43.221871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.221847    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg"
Nov 03 08:23:43.221955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.221899    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg" event={"ID":"afa12a2a-bfc8-46f0-9647-1ee6f00447d4","Type":"ContainerDied","Data":"d925beace2811dfa5718f044e5752bfea2ecdef76d092941ccef553029bd0cef"}
Nov 03 08:23:43.221955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.221939    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d925beace2811dfa5718f044e5752bfea2ecdef76d092941ccef553029bd0cef"
Nov 03 08:23:43.223516 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.223497    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046740e4-g9hl5" event={"ID":"20d8ac2c-99c2-4125-ae50-cd90c6a78609","Type":"ContainerDied","Data":"3f2bf9ad7cd48f4b64cef5d0c556df665a4f438df9c96c7b59386a0706ab3902"}
Nov 03 08:23:43.223612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.223528    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046740e4-g9hl5"
Nov 03 08:23:43.223612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.223531    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f2bf9ad7cd48f4b64cef5d0c556df665a4f438df9c96c7b59386a0706ab3902"
Nov 03 08:23:43.250612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.250592    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20d8ac2c-99c2-4125-ae50-cd90c6a78609-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d8ac2c-99c2-4125-ae50-cd90c6a78609-maas-api-xks-ca\") pod \"20d8ac2c-99c2-4125-ae50-cd90c6a78609\" (UID: \"20d8ac2c-99c2-4125-ae50-cd90c6a78609\") "
Nov 03 08:23:43.250704 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.250659    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afa12a2a-bfc8-46f0-9647-1ee6f00447d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa12a2a-bfc8-46f0-9647-1ee6f00447d4-maas-api-openshift-ca\") pod \"afa12a2a-bfc8-46f0-9647-1ee6f00447d4\" (UID: \"afa12a2a-bfc8-46f0-9647-1ee6f00447d4\") "
Nov 03 08:23:43.250704 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.250685    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afa12a2a-bfc8-46f0-9647-1ee6f00447d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa12a2a-bfc8-46f0-9647-1ee6f00447d4-maas-api-xks-ca\") pod \"afa12a2a-bfc8-46f0-9647-1ee6f00447d4\" (UID: \"afa12a2a-bfc8-46f0-9647-1ee6f00447d4\") "
Nov 03 08:23:43.250829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.250734    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4250e197-3658-4c5d-8130-35878b72827d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4250e197-3658-4c5d-8130-35878b72827d-maas-api-openshift-ca\") pod \"4250e197-3658-4c5d-8130-35878b72827d\" (UID: \"4250e197-3658-4c5d-8130-35878b72827d\") "
Nov 03 08:23:43.250894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.250855    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4250e197-3658-4c5d-8130-35878b72827d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4250e197-3658-4c5d-8130-35878b72827d-maas-api-xks-ca\") pod \"4250e197-3658-4c5d-8130-35878b72827d\" (UID: \"4250e197-3658-4c5d-8130-35878b72827d\") "
Nov 03 08:23:43.250955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.250898    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afa12a2a-bfc8-46f0-9647-1ee6f00447d4-kube-api-access-dgpxw\" (UniqueName: \"kubernetes.io/projected/afa12a2a-bfc8-46f0-9647-1ee6f00447d4-kube-api-access-dgpxw\") pod \"afa12a2a-bfc8-46f0-9647-1ee6f00447d4\" (UID: \"afa12a2a-bfc8-46f0-9647-1ee6f00447d4\") "
Nov 03 08:23:43.250955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.250932    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4250e197-3658-4c5d-8130-35878b72827d-kube-api-access-886zs\" (UniqueName: \"kubernetes.io/projected/4250e197-3658-4c5d-8130-35878b72827d-kube-api-access-886zs\") pod \"4250e197-3658-4c5d-8130-35878b72827d\" (UID: \"4250e197-3658-4c5d-8130-35878b72827d\") "
Nov 03 08:23:43.251127 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.251104    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afa12a2a-bfc8-46f0-9647-1ee6f00447d4-maas-api-openshift-ca" pod "afa12a2a-bfc8-46f0-9647-1ee6f00447d4" (UID: "afa12a2a-bfc8-46f0-9647-1ee6f00447d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:43.251189 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.251133    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4250e197-3658-4c5d-8130-35878b72827d-maas-api-openshift-ca" pod "4250e197-3658-4c5d-8130-35878b72827d" (UID: "4250e197-3658-4c5d-8130-35878b72827d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:43.251351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.251328    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d8ac2c-99c2-4125-ae50-cd90c6a78609-kube-api-access-5bhgp\" (UniqueName: \"kubernetes.io/projected/20d8ac2c-99c2-4125-ae50-cd90c6a78609-kube-api-access-5bhgp\") pod \"20d8ac2c-99c2-4125-ae50-cd90c6a78609\" (UID: \"20d8ac2c-99c2-4125-ae50-cd90c6a78609\") "
Nov 03 08:23:43.251427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.251373    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20d8ac2c-99c2-4125-ae50-cd90c6a78609-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d8ac2c-99c2-4125-ae50-cd90c6a78609-maas-api-openshift-ca\") pod \"20d8ac2c-99c2-4125-ae50-cd90c6a78609\" (UID: \"20d8ac2c-99c2-4125-ae50-cd90c6a78609\") "
Nov 03 08:23:43.252508 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.251857    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d8ac2c-99c2-4125-ae50-cd90c6a78609-maas-api-openshift-ca" pod "20d8ac2c-99c2-4125-ae50-cd90c6a78609" (UID: "20d8ac2c-99c2-4125-ae50-cd90c6a78609"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:43.252508 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.252443    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa12a2a-bfc8-46f0-9647-1ee6f00447d4-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:43.252508 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.252468    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4250e197-3658-4c5d-8130-35878b72827d-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:43.252508 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.252485    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d8ac2c-99c2-4125-ae50-cd90c6a78609-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:43.253151 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.253113    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20d8ac2c-99c2-4125-ae50-cd90c6a78609-maas-api-xks-ca" pod "20d8ac2c-99c2-4125-ae50-cd90c6a78609" (UID: "20d8ac2c-99c2-4125-ae50-cd90c6a78609"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:43.253503 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.253462    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afa12a2a-bfc8-46f0-9647-1ee6f00447d4-maas-api-xks-ca" pod "afa12a2a-bfc8-46f0-9647-1ee6f00447d4" (UID: "afa12a2a-bfc8-46f0-9647-1ee6f00447d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:43.253605 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.253540    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4250e197-3658-4c5d-8130-35878b72827d-maas-api-xks-ca" pod "4250e197-3658-4c5d-8130-35878b72827d" (UID: "4250e197-3658-4c5d-8130-35878b72827d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:43.253683 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.253597    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa12a2a-bfc8-46f0-9647-1ee6f00447d4-kube-api-access-dgpxw" pod "afa12a2a-bfc8-46f0-9647-1ee6f00447d4" (UID: "afa12a2a-bfc8-46f0-9647-1ee6f00447d4"). InnerVolumeSpecName "kube-api-access-dgpxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:43.254045 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.254025    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d8ac2c-99c2-4125-ae50-cd90c6a78609-kube-api-access-5bhgp" pod "20d8ac2c-99c2-4125-ae50-cd90c6a78609" (UID: "20d8ac2c-99c2-4125-ae50-cd90c6a78609"). InnerVolumeSpecName "kube-api-access-5bhgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:43.254132 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.254089    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4250e197-3658-4c5d-8130-35878b72827d-kube-api-access-886zs" pod "4250e197-3658-4c5d-8130-35878b72827d" (UID: "4250e197-3658-4c5d-8130-35878b72827d"). InnerVolumeSpecName "kube-api-access-886zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:43.355115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.355078    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bhgp\" (UniqueName: \"kubernetes.io/projected/20d8ac2c-99c2-4125-ae50-cd90c6a78609-kube-api-access-5bhgp\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:43.355115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.355117    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d8ac2c-99c2-4125-ae50-cd90c6a78609-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:43.355326 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.355133    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa12a2a-bfc8-46f0-9647-1ee6f00447d4-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:43.355326 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.355147    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4250e197-3658-4c5d-8130-35878b72827d-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:43.355326 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.355161    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgpxw\" (UniqueName: \"kubernetes.io/projected/afa12a2a-bfc8-46f0-9647-1ee6f00447d4-kube-api-access-dgpxw\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:43.355326 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.355175    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-886zs\" (UniqueName: \"kubernetes.io/projected/4250e197-3658-4c5d-8130-35878b72827d-kube-api-access-886zs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:43.916970 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:43.916937    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:23:46.817856 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.817806    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq"]
Nov 03 08:23:46.821399 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.821370    5070 cpu_manager.go:414] "containerMap: removing container" podUID="20d8ac2c-99c2-4125-ae50-cd90c6a78609" containerName="revoke-keys"
Nov 03 08:23:46.823507 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.823485    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d8ac2c-99c2-4125-ae50-cd90c6a78609" containerName="revoke-keys"
Nov 03 08:23:46.823746 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.823730    5070 cpu_manager.go:414] "containerMap: removing container" podUID="4250e197-3658-4c5d-8130-35878b72827d" containerName="revoke-keys"
Nov 03 08:23:46.823887 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.823872    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4250e197-3658-4c5d-8130-35878b72827d" containerName="revoke-keys"
Nov 03 08:23:46.824000 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.823988    5070 cpu_manager.go:414] "containerMap: removing container" podUID="afa12a2a-bfc8-46f0-9647-1ee6f00447d4" containerName="revoke-keys"
Nov 03 08:23:46.824092 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.824080    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa12a2a-bfc8-46f0-9647-1ee6f00447d4" containerName="revoke-keys"
Nov 03 08:23:46.824209 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.824198    5070 cpu_manager.go:414] "containerMap: removing container" podUID="920822a5-777d-42f1-af1f-d0cff4eb1e52" containerName="revoke-keys"
Nov 03 08:23:46.824300 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.824288    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="920822a5-777d-42f1-af1f-d0cff4eb1e52" containerName="revoke-keys"
Nov 03 08:23:46.824581 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.824562    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa12a2a-bfc8-46f0-9647-1ee6f00447d4" containerName="revoke-keys"
Nov 03 08:23:46.824716 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.824703    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="920822a5-777d-42f1-af1f-d0cff4eb1e52" containerName="revoke-keys"
Nov 03 08:23:46.824853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.824840    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d8ac2c-99c2-4125-ae50-cd90c6a78609" containerName="revoke-keys"
Nov 03 08:23:46.824974 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.824962    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="4250e197-3658-4c5d-8130-35878b72827d" containerName="revoke-keys"
Nov 03 08:23:46.829694 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.829674    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq"
Nov 03 08:23:46.835615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.835583    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:23:46.836439 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.836421    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:23:46.836665 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.836651    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:23:46.837704 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.837687    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq"]
Nov 03 08:23:46.841223 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.839564    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:23:46.846130 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.845963    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz"]
Nov 03 08:23:46.857657 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.857630    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz"]
Nov 03 08:23:46.857958 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.857942    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz"
Nov 03 08:23:46.908396 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.908343    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shwmn\" (UniqueName: \"kubernetes.io/projected/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7-kube-api-access-shwmn\") pod \"maas-api-revoke-sub-e376881b-5w7cq\" (UID: \"6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq"
Nov 03 08:23:46.908615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.908427    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27376302-17ef-4469-9b08-d882df27e7de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e09976b-8xqzz\" (UID: \"27376302-17ef-4469-9b08-d882df27e7de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz"
Nov 03 08:23:46.908615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.908482    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsp6n\" (UniqueName: \"kubernetes.io/projected/27376302-17ef-4469-9b08-d882df27e7de-kube-api-access-dsp6n\") pod \"maas-api-revoke-sub-9e09976b-8xqzz\" (UID: \"27376302-17ef-4469-9b08-d882df27e7de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz"
Nov 03 08:23:46.908854 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.908629    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27376302-17ef-4469-9b08-d882df27e7de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e09976b-8xqzz\" (UID: \"27376302-17ef-4469-9b08-d882df27e7de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz"
Nov 03 08:23:46.908854 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.908719    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e376881b-5w7cq\" (UID: \"6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq"
Nov 03 08:23:46.908978 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:46.908873    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e376881b-5w7cq\" (UID: \"6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq"
Nov 03 08:23:47.045786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:47.040175    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27376302-17ef-4469-9b08-d882df27e7de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e09976b-8xqzz\" (UID: \"27376302-17ef-4469-9b08-d882df27e7de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz"
Nov 03 08:23:47.045786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:47.040285    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e376881b-5w7cq\" (UID: \"6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq"
Nov 03 08:23:47.045786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:47.040440    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e376881b-5w7cq\" (UID: \"6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq"
Nov 03 08:23:47.045786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:47.040534    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shwmn\" (UniqueName: \"kubernetes.io/projected/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7-kube-api-access-shwmn\") pod \"maas-api-revoke-sub-e376881b-5w7cq\" (UID: \"6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq"
Nov 03 08:23:47.045786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:47.040605    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27376302-17ef-4469-9b08-d882df27e7de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e09976b-8xqzz\" (UID: \"27376302-17ef-4469-9b08-d882df27e7de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz"
Nov 03 08:23:47.045786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:47.040664    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsp6n\" (UniqueName: \"kubernetes.io/projected/27376302-17ef-4469-9b08-d882df27e7de-kube-api-access-dsp6n\") pod \"maas-api-revoke-sub-9e09976b-8xqzz\" (UID: \"27376302-17ef-4469-9b08-d882df27e7de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz"
Nov 03 08:23:47.051986 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:47.051898    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27376302-17ef-4469-9b08-d882df27e7de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e09976b-8xqzz\" (UID: \"27376302-17ef-4469-9b08-d882df27e7de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz"
Nov 03 08:23:47.053546 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:47.052986    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e376881b-5w7cq\" (UID: \"6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq"
Nov 03 08:23:47.056459 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:47.056392    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e376881b-5w7cq\" (UID: \"6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq"
Nov 03 08:23:47.074261 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:47.074156    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shwmn\" (UniqueName: \"kubernetes.io/projected/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7-kube-api-access-shwmn\") pod \"maas-api-revoke-sub-e376881b-5w7cq\" (UID: \"6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq"
Nov 03 08:23:47.078729 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:47.078685    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsp6n\" (UniqueName: \"kubernetes.io/projected/27376302-17ef-4469-9b08-d882df27e7de-kube-api-access-dsp6n\") pod \"maas-api-revoke-sub-9e09976b-8xqzz\" (UID: \"27376302-17ef-4469-9b08-d882df27e7de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz"
Nov 03 08:23:47.080106 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:47.080081    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27376302-17ef-4469-9b08-d882df27e7de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e09976b-8xqzz\" (UID: \"27376302-17ef-4469-9b08-d882df27e7de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz"
Nov 03 08:23:47.170941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:47.170912    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq"
Nov 03 08:23:47.200547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:47.200516    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz"
Nov 03 08:23:47.480030 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:47.479921    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq"]
Nov 03 08:23:47.485641 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:23:47.485598    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0e4de6_c3e9_45f3_8542_91ed6f0f0bf7.slice/crio-76cc6382a5dd59d3aa1b305360e3e0eb5fdbe8cf78cdbf890ade2c4c1968e6d6 WatchSource:0}: Error finding container 76cc6382a5dd59d3aa1b305360e3e0eb5fdbe8cf78cdbf890ade2c4c1968e6d6: Status 404 returned error can't find the container with id 76cc6382a5dd59d3aa1b305360e3e0eb5fdbe8cf78cdbf890ade2c4c1968e6d6
Nov 03 08:23:47.512502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:47.512413    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz"]
Nov 03 08:23:47.515523 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:23:47.515456    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27376302_17ef_4469_9b08_d882df27e7de.slice/crio-dbb23a4bf91efa045d797e3002e6fccb7eed2c01dbaf7cfa24a4bd63bf65a0b8 WatchSource:0}: Error finding container dbb23a4bf91efa045d797e3002e6fccb7eed2c01dbaf7cfa24a4bd63bf65a0b8: Status 404 returned error can't find the container with id dbb23a4bf91efa045d797e3002e6fccb7eed2c01dbaf7cfa24a4bd63bf65a0b8
Nov 03 08:23:48.188976 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:48.188937    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb" podUID="d449195e-0ed5-4c8e-ac1b-3242198699f7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:23:48.189401 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:48.189077    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m" podUID="cb3722fc-983b-4b33-9269-8e559475aff5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:23:48.189401 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:48.189079    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn" podUID="063e3656-6675-4336-a3c1-cf96b568be28" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:23:48.445387 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:48.445334    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz" event={"ID":"27376302-17ef-4469-9b08-d882df27e7de","Type":"ContainerStarted","Data":"26f562bf40571169c9c466d011e4e469e67f0b9f73579faf7a7c3c764e05e69d"}
Nov 03 08:23:48.445583 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:48.445415    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz" event={"ID":"27376302-17ef-4469-9b08-d882df27e7de","Type":"ContainerStarted","Data":"dbb23a4bf91efa045d797e3002e6fccb7eed2c01dbaf7cfa24a4bd63bf65a0b8"}
Nov 03 08:23:48.447359 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:48.447323    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq" event={"ID":"6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7","Type":"ContainerStarted","Data":"d481e7e7b611192ca40c18e95f5272f592b515079aa1f8bb09e9a6ab6ad9def3"}
Nov 03 08:23:48.447584 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:48.447555    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq" event={"ID":"6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7","Type":"ContainerStarted","Data":"76cc6382a5dd59d3aa1b305360e3e0eb5fdbe8cf78cdbf890ade2c4c1968e6d6"}
Nov 03 08:23:48.466148 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:48.466103    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz" podStartSLOduration=2.466089861 podStartE2EDuration="2.466089861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:23:48.460392165 +0000 UTC m=+1826.574856098" watchObservedRunningTime="2026-11-03 08:23:48.466089861 +0000 UTC m=+1826.580553815"
Nov 03 08:23:48.481652 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:48.481607    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq" podStartSLOduration=2.4815949330000002 podStartE2EDuration="2.481594933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:23:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:23:48.478677496 +0000 UTC m=+1826.593141429" watchObservedRunningTime="2026-11-03 08:23:48.481594933 +0000 UTC m=+1826.596058864"
Nov 03 08:23:49.402560 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.402526    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m"
Nov 03 08:23:49.406268 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.406244    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb"
Nov 03 08:23:49.494272 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.494172    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d449195e-0ed5-4c8e-ac1b-3242198699f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d449195e-0ed5-4c8e-ac1b-3242198699f7-plugins-config-volume\") pod \"d449195e-0ed5-4c8e-ac1b-3242198699f7\" (UID: \"d449195e-0ed5-4c8e-ac1b-3242198699f7\") "
Nov 03 08:23:49.496047 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.494497    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb3722fc-983b-4b33-9269-8e559475aff5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3722fc-983b-4b33-9269-8e559475aff5-plugins-config-volume\") pod \"cb3722fc-983b-4b33-9269-8e559475aff5\" (UID: \"cb3722fc-983b-4b33-9269-8e559475aff5\") "
Nov 03 08:23:49.496047 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.494537    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3722fc-983b-4b33-9269-8e559475aff5-kube-api-access-kdhvz\" (UniqueName: \"kubernetes.io/projected/cb3722fc-983b-4b33-9269-8e559475aff5-kube-api-access-kdhvz\") pod \"cb3722fc-983b-4b33-9269-8e559475aff5\" (UID: \"cb3722fc-983b-4b33-9269-8e559475aff5\") "
Nov 03 08:23:49.496047 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.494626    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d449195e-0ed5-4c8e-ac1b-3242198699f7-kube-api-access-mfsd5\" (UniqueName: \"kubernetes.io/projected/d449195e-0ed5-4c8e-ac1b-3242198699f7-kube-api-access-mfsd5\") pod \"d449195e-0ed5-4c8e-ac1b-3242198699f7\" (UID: \"d449195e-0ed5-4c8e-ac1b-3242198699f7\") "
Nov 03 08:23:49.519824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.519744    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d449195e-0ed5-4c8e-ac1b-3242198699f7-plugins-config-volume" pod "d449195e-0ed5-4c8e-ac1b-3242198699f7" (UID: "d449195e-0ed5-4c8e-ac1b-3242198699f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:49.522573 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.522539    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3722fc-983b-4b33-9269-8e559475aff5-plugins-config-volume" pod "cb3722fc-983b-4b33-9269-8e559475aff5" (UID: "cb3722fc-983b-4b33-9269-8e559475aff5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:49.522790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.522770    5070 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb3722fc-983b-4b33-9269-8e559475aff5-plugins-config-volume\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:49.522879 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.522796    5070 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d449195e-0ed5-4c8e-ac1b-3242198699f7-plugins-config-volume\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:49.526860 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.526828    5070 generic.go:356] "Generic (PLEG): container finished" podID="27376302-17ef-4469-9b08-d882df27e7de" containerID="26f562bf40571169c9c466d011e4e469e67f0b9f73579faf7a7c3c764e05e69d" exitCode=0
Nov 03 08:23:49.527025 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.527004    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz" event={"ID":"27376302-17ef-4469-9b08-d882df27e7de","Type":"ContainerDied","Data":"26f562bf40571169c9c466d011e4e469e67f0b9f73579faf7a7c3c764e05e69d"}
Nov 03 08:23:49.529440 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.529408    5070 generic.go:356] "Generic (PLEG): container finished" podID="cb3722fc-983b-4b33-9269-8e559475aff5" containerID="84944f5477498a6baef9b706c16bad14f7b62a33ae6c8ab56da34661d69850c2" exitCode=0
Nov 03 08:23:49.529623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.529603    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m" event={"ID":"cb3722fc-983b-4b33-9269-8e559475aff5","Type":"ContainerDied","Data":"84944f5477498a6baef9b706c16bad14f7b62a33ae6c8ab56da34661d69850c2"}
Nov 03 08:23:49.529681 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.529654    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m" event={"ID":"cb3722fc-983b-4b33-9269-8e559475aff5","Type":"ContainerDied","Data":"9fc9fdb6f59d740369d9e95debabbde69b90a586d2224f5d4c383706f48ed2c7"}
Nov 03 08:23:49.529731 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.529686    5070 scope.go:122] "RemoveContainer" containerID="84944f5477498a6baef9b706c16bad14f7b62a33ae6c8ab56da34661d69850c2"
Nov 03 08:23:49.529892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.529875    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m"
Nov 03 08:23:49.532339 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.532295    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d449195e-0ed5-4c8e-ac1b-3242198699f7-kube-api-access-mfsd5" pod "d449195e-0ed5-4c8e-ac1b-3242198699f7" (UID: "d449195e-0ed5-4c8e-ac1b-3242198699f7"). InnerVolumeSpecName "kube-api-access-mfsd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:49.533321 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.533296    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3722fc-983b-4b33-9269-8e559475aff5-kube-api-access-kdhvz" pod "cb3722fc-983b-4b33-9269-8e559475aff5" (UID: "cb3722fc-983b-4b33-9269-8e559475aff5"). InnerVolumeSpecName "kube-api-access-kdhvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:49.536938 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.536915    5070 generic.go:356] "Generic (PLEG): container finished" podID="d449195e-0ed5-4c8e-ac1b-3242198699f7" containerID="5af597feaa1c3de87495549902e7b6a22b74cecb6a7d6392963fd11a76083a22" exitCode=0
Nov 03 08:23:49.537049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.536980    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb" event={"ID":"d449195e-0ed5-4c8e-ac1b-3242198699f7","Type":"ContainerDied","Data":"5af597feaa1c3de87495549902e7b6a22b74cecb6a7d6392963fd11a76083a22"}
Nov 03 08:23:49.537049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.537010    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb" event={"ID":"d449195e-0ed5-4c8e-ac1b-3242198699f7","Type":"ContainerDied","Data":"0782d2243d4471d30e8e640e4f04c705da68325525558dc22ab9be2eab1dddf7"}
Nov 03 08:23:49.537157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.537070    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb"
Nov 03 08:23:49.541155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.540655    5070 generic.go:356] "Generic (PLEG): container finished" podID="6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7" containerID="d481e7e7b611192ca40c18e95f5272f592b515079aa1f8bb09e9a6ab6ad9def3" exitCode=0
Nov 03 08:23:49.541155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.540834    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq" event={"ID":"6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7","Type":"ContainerDied","Data":"d481e7e7b611192ca40c18e95f5272f592b515079aa1f8bb09e9a6ab6ad9def3"}
Nov 03 08:23:49.565150 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.565120    5070 scope.go:122] "RemoveContainer" containerID="84944f5477498a6baef9b706c16bad14f7b62a33ae6c8ab56da34661d69850c2"
Nov 03 08:23:49.565485 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:23:49.565467    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84944f5477498a6baef9b706c16bad14f7b62a33ae6c8ab56da34661d69850c2\": container with ID starting with 84944f5477498a6baef9b706c16bad14f7b62a33ae6c8ab56da34661d69850c2 not found: ID does not exist" containerID="84944f5477498a6baef9b706c16bad14f7b62a33ae6c8ab56da34661d69850c2"
Nov 03 08:23:49.565568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.565500    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84944f5477498a6baef9b706c16bad14f7b62a33ae6c8ab56da34661d69850c2"} err="failed to get container status \"84944f5477498a6baef9b706c16bad14f7b62a33ae6c8ab56da34661d69850c2\": rpc error: code = NotFound desc = could not find container \"84944f5477498a6baef9b706c16bad14f7b62a33ae6c8ab56da34661d69850c2\": container with ID starting with 84944f5477498a6baef9b706c16bad14f7b62a33ae6c8ab56da34661d69850c2 not found: ID does not exist"
Nov 03 08:23:49.565568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.565525    5070 scope.go:122] "RemoveContainer" containerID="5af597feaa1c3de87495549902e7b6a22b74cecb6a7d6392963fd11a76083a22"
Nov 03 08:23:49.580929 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.580906    5070 scope.go:122] "RemoveContainer" containerID="5af597feaa1c3de87495549902e7b6a22b74cecb6a7d6392963fd11a76083a22"
Nov 03 08:23:49.581474 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:23:49.581442    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af597feaa1c3de87495549902e7b6a22b74cecb6a7d6392963fd11a76083a22\": container with ID starting with 5af597feaa1c3de87495549902e7b6a22b74cecb6a7d6392963fd11a76083a22 not found: ID does not exist" containerID="5af597feaa1c3de87495549902e7b6a22b74cecb6a7d6392963fd11a76083a22"
Nov 03 08:23:49.581621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.581483    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af597feaa1c3de87495549902e7b6a22b74cecb6a7d6392963fd11a76083a22"} err="failed to get container status \"5af597feaa1c3de87495549902e7b6a22b74cecb6a7d6392963fd11a76083a22\": rpc error: code = NotFound desc = could not find container \"5af597feaa1c3de87495549902e7b6a22b74cecb6a7d6392963fd11a76083a22\": container with ID starting with 5af597feaa1c3de87495549902e7b6a22b74cecb6a7d6392963fd11a76083a22 not found: ID does not exist"
Nov 03 08:23:49.590240 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.590216    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb"]
Nov 03 08:23:49.595583 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.595565    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-15a633-5c45f88d4c-wsdpb"]
Nov 03 08:23:49.625065 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.625035    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdhvz\" (UniqueName: \"kubernetes.io/projected/cb3722fc-983b-4b33-9269-8e559475aff5-kube-api-access-kdhvz\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:49.625065 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.625060    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfsd5\" (UniqueName: \"kubernetes.io/projected/d449195e-0ed5-4c8e-ac1b-3242198699f7-kube-api-access-mfsd5\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:49.857294 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.857266    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m"]
Nov 03 08:23:49.864696 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:49.864669    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-15a633-594b7b75cc-smg5m"]
Nov 03 08:23:50.029068 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:50.029038    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb3722fc-983b-4b33-9269-8e559475aff5" path="/var/lib/kubelet/pods/cb3722fc-983b-4b33-9269-8e559475aff5/volumes"
Nov 03 08:23:50.029564 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:50.029550    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d449195e-0ed5-4c8e-ac1b-3242198699f7" path="/var/lib/kubelet/pods/d449195e-0ed5-4c8e-ac1b-3242198699f7/volumes"
Nov 03 08:23:50.832447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:50.832315    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq"
Nov 03 08:23:50.835543 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:50.835516    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz"
Nov 03 08:23:50.957598 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:50.957561    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27376302-17ef-4469-9b08-d882df27e7de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27376302-17ef-4469-9b08-d882df27e7de-maas-api-openshift-ca\") pod \"27376302-17ef-4469-9b08-d882df27e7de\" (UID: \"27376302-17ef-4469-9b08-d882df27e7de\") "
Nov 03 08:23:50.957819 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:50.957658    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7-maas-api-openshift-ca\") pod \"6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7\" (UID: \"6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7\") "
Nov 03 08:23:50.957819 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:50.957707    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7-kube-api-access-shwmn\" (UniqueName: \"kubernetes.io/projected/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7-kube-api-access-shwmn\") pod \"6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7\" (UID: \"6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7\") "
Nov 03 08:23:50.957819 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:50.957808    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7-maas-api-xks-ca\") pod \"6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7\" (UID: \"6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7\") "
Nov 03 08:23:50.958017 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:50.957980    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27376302-17ef-4469-9b08-d882df27e7de-maas-api-openshift-ca" pod "27376302-17ef-4469-9b08-d882df27e7de" (UID: "27376302-17ef-4469-9b08-d882df27e7de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:50.958147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:50.958121    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7-maas-api-openshift-ca" pod "6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7" (UID: "6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:50.958228 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:50.958153    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27376302-17ef-4469-9b08-d882df27e7de-kube-api-access-dsp6n\" (UniqueName: \"kubernetes.io/projected/27376302-17ef-4469-9b08-d882df27e7de-kube-api-access-dsp6n\") pod \"27376302-17ef-4469-9b08-d882df27e7de\" (UID: \"27376302-17ef-4469-9b08-d882df27e7de\") "
Nov 03 08:23:50.958228 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:50.958187    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27376302-17ef-4469-9b08-d882df27e7de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27376302-17ef-4469-9b08-d882df27e7de-maas-api-xks-ca\") pod \"27376302-17ef-4469-9b08-d882df27e7de\" (UID: \"27376302-17ef-4469-9b08-d882df27e7de\") "
Nov 03 08:23:50.959100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:50.959075    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27376302-17ef-4469-9b08-d882df27e7de-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:50.959328 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:50.959310    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:50.960579 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:50.960544    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7-maas-api-xks-ca" pod "6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7" (UID: "6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:50.960677 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:50.960596    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7-kube-api-access-shwmn" pod "6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7" (UID: "6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7"). InnerVolumeSpecName "kube-api-access-shwmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:50.960857 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:50.960835    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27376302-17ef-4469-9b08-d882df27e7de-kube-api-access-dsp6n" pod "27376302-17ef-4469-9b08-d882df27e7de" (UID: "27376302-17ef-4469-9b08-d882df27e7de"). InnerVolumeSpecName "kube-api-access-dsp6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:50.961867 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:50.961838    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27376302-17ef-4469-9b08-d882df27e7de-maas-api-xks-ca" pod "27376302-17ef-4469-9b08-d882df27e7de" (UID: "27376302-17ef-4469-9b08-d882df27e7de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:23:51.060653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:51.060578    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsp6n\" (UniqueName: \"kubernetes.io/projected/27376302-17ef-4469-9b08-d882df27e7de-kube-api-access-dsp6n\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:51.060653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:51.060609    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27376302-17ef-4469-9b08-d882df27e7de-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:51.060653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:51.060618    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shwmn\" (UniqueName: \"kubernetes.io/projected/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7-kube-api-access-shwmn\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:51.060653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:51.060627    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:51.567147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:51.567119    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz"
Nov 03 08:23:51.567300 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:51.567162    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz" event={"ID":"27376302-17ef-4469-9b08-d882df27e7de","Type":"ContainerDied","Data":"dbb23a4bf91efa045d797e3002e6fccb7eed2c01dbaf7cfa24a4bd63bf65a0b8"}
Nov 03 08:23:51.567300 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:51.567209    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbb23a4bf91efa045d797e3002e6fccb7eed2c01dbaf7cfa24a4bd63bf65a0b8"
Nov 03 08:23:51.568955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:51.568931    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq"
Nov 03 08:23:51.569061 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:51.568968    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq" event={"ID":"6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7","Type":"ContainerDied","Data":"76cc6382a5dd59d3aa1b305360e3e0eb5fdbe8cf78cdbf890ade2c4c1968e6d6"}
Nov 03 08:23:51.569061 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:51.569014    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76cc6382a5dd59d3aa1b305360e3e0eb5fdbe8cf78cdbf890ade2c4c1968e6d6"
Nov 03 08:23:52.088950 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:23:52.088920    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:23:56.158253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:56.158217    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts"]
Nov 03 08:23:56.163094 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:56.163073    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-e0f576-br9ts"]
Nov 03 08:23:56.662526 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:56.662483    5070 generic.go:356] "Generic (PLEG): container finished" podID="063e3656-6675-4336-a3c1-cf96b568be28" containerID="89d6a74919c8b739e8ca254ff8e9552e65775ceab1360487a2b6a05bb8333217" exitCode=0
Nov 03 08:23:56.662706 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:56.662618    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn" event={"ID":"063e3656-6675-4336-a3c1-cf96b568be28","Type":"ContainerDied","Data":"89d6a74919c8b739e8ca254ff8e9552e65775ceab1360487a2b6a05bb8333217"}
Nov 03 08:23:56.754711 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:56.754675    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48"]
Nov 03 08:23:56.763752 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:56.763718    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-15a633-jxl48"]
Nov 03 08:23:56.827420 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:56.827389    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn"
Nov 03 08:23:57.055263 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:57.055186    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/063e3656-6675-4336-a3c1-cf96b568be28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/063e3656-6675-4336-a3c1-cf96b568be28-plugins-config-volume\") pod \"063e3656-6675-4336-a3c1-cf96b568be28\" (UID: \"063e3656-6675-4336-a3c1-cf96b568be28\") "
Nov 03 08:23:57.055408 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:57.055393    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/063e3656-6675-4336-a3c1-cf96b568be28-kube-api-access-bhr4p\" (UniqueName: \"kubernetes.io/projected/063e3656-6675-4336-a3c1-cf96b568be28-kube-api-access-bhr4p\") pod \"063e3656-6675-4336-a3c1-cf96b568be28\" (UID: \"063e3656-6675-4336-a3c1-cf96b568be28\") "
Nov 03 08:23:57.055572 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:57.055549    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/063e3656-6675-4336-a3c1-cf96b568be28-plugins-config-volume" pod "063e3656-6675-4336-a3c1-cf96b568be28" (UID: "063e3656-6675-4336-a3c1-cf96b568be28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:23:57.055894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:57.055877    5070 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/063e3656-6675-4336-a3c1-cf96b568be28-plugins-config-volume\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:57.057585 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:57.057561    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/063e3656-6675-4336-a3c1-cf96b568be28-kube-api-access-bhr4p" pod "063e3656-6675-4336-a3c1-cf96b568be28" (UID: "063e3656-6675-4336-a3c1-cf96b568be28"). InnerVolumeSpecName "kube-api-access-bhr4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:23:57.157502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:57.157472    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhr4p\" (UniqueName: \"kubernetes.io/projected/063e3656-6675-4336-a3c1-cf96b568be28-kube-api-access-bhr4p\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:23:57.675206 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:57.675177    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn"
Nov 03 08:23:57.675596 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:57.675214    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn" event={"ID":"063e3656-6675-4336-a3c1-cf96b568be28","Type":"ContainerDied","Data":"c367495c8ef0d05930237ef4df513308e0bb2ed1ef76363bdd28ac0ca9d31916"}
Nov 03 08:23:57.675596 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:57.675277    5070 scope.go:122] "RemoveContainer" containerID="89d6a74919c8b739e8ca254ff8e9552e65775ceab1360487a2b6a05bb8333217"
Nov 03 08:23:57.699134 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:57.699111    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn"]
Nov 03 08:23:57.703868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:57.703845    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-76ef13-5b5fc8489f-v29jn"]
Nov 03 08:23:58.030922 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:58.030837    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="063e3656-6675-4336-a3c1-cf96b568be28" path="/var/lib/kubelet/pods/063e3656-6675-4336-a3c1-cf96b568be28/volumes"
Nov 03 08:23:58.032695 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:58.032670    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61b57c27-f0e7-4864-b9f1-fa221b7d7bec" path="/var/lib/kubelet/pods/61b57c27-f0e7-4864-b9f1-fa221b7d7bec/volumes"
Nov 03 08:23:58.033817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:23:58.033795    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="920822a5-777d-42f1-af1f-d0cff4eb1e52" path="/var/lib/kubelet/pods/920822a5-777d-42f1-af1f-d0cff4eb1e52/volumes"
Nov 03 08:24:02.653671 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:02.653637    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"]
Nov 03 08:24:02.659643 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:02.659614    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"]
Nov 03 08:24:02.660563 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:02.660531    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv" podUID="198146ed-665b-4801-8396-c980ee6ff4d4" containerName="istio-proxy" containerID="cri-o://aaba34aa082b19fa1fe5fd4bf0982a15a6b4e7a9fd5623ca022458171a88a036" gracePeriod=30
Nov 03 08:24:02.668123 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:02.668102    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"]
Nov 03 08:24:02.668483 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:02.668456    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm" podUID="be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" containerName="istio-proxy" containerID="cri-o://e4d3100066a6b2b6e4b419aef7f8dd47e9de612dd61cc0a1256e740e23b50e46" gracePeriod=30
Nov 03 08:24:02.678668 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:02.678642    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"]
Nov 03 08:24:02.678907 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:02.678879    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt" podUID="d4174a67-8cbf-452f-b382-8efd46377534" containerName="istio-proxy" containerID="cri-o://a25db031f450eb71ce100592f948cee1701b87545c849891ee22aacf0462587f" gracePeriod=30
Nov 03 08:24:02.732686 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:02.732641    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy" containerID="cri-o://af8c47943e052cd3b42beaabdc4afa3e932d82b9358e85fdb77720ea38f9921c" gracePeriod=30
Nov 03 08:24:03.247708 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.247663    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg"]
Nov 03 08:24:03.259440 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.259399    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9b559e-bbtwg"]
Nov 03 08:24:03.291424 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.291390    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll"]
Nov 03 08:24:03.292632 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.292608    5070 cpu_manager.go:414] "containerMap: removing container" podUID="cb3722fc-983b-4b33-9269-8e559475aff5" containerName="payload-processing"
Nov 03 08:24:03.292709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.292636    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3722fc-983b-4b33-9269-8e559475aff5" containerName="payload-processing"
Nov 03 08:24:03.292709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.292698    5070 cpu_manager.go:414] "containerMap: removing container" podUID="063e3656-6675-4336-a3c1-cf96b568be28" containerName="payload-processing"
Nov 03 08:24:03.292709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.292706    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="063e3656-6675-4336-a3c1-cf96b568be28" containerName="payload-processing"
Nov 03 08:24:03.292885 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.292720    5070 cpu_manager.go:414] "containerMap: removing container" podUID="27376302-17ef-4469-9b08-d882df27e7de" containerName="revoke-keys"
Nov 03 08:24:03.292885 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.292727    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27376302-17ef-4469-9b08-d882df27e7de" containerName="revoke-keys"
Nov 03 08:24:03.292885 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.292746    5070 cpu_manager.go:414] "containerMap: removing container" podUID="d449195e-0ed5-4c8e-ac1b-3242198699f7" containerName="payload-pre-processing"
Nov 03 08:24:03.292885 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.292769    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d449195e-0ed5-4c8e-ac1b-3242198699f7" containerName="payload-pre-processing"
Nov 03 08:24:03.292885 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.292797    5070 cpu_manager.go:414] "containerMap: removing container" podUID="6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7" containerName="revoke-keys"
Nov 03 08:24:03.292885 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.292805    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7" containerName="revoke-keys"
Nov 03 08:24:03.293100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.293003    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="063e3656-6675-4336-a3c1-cf96b568be28" containerName="payload-processing"
Nov 03 08:24:03.293100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.293019    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="27376302-17ef-4469-9b08-d882df27e7de" containerName="revoke-keys"
Nov 03 08:24:03.293100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.293029    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="d449195e-0ed5-4c8e-ac1b-3242198699f7" containerName="payload-pre-processing"
Nov 03 08:24:03.293100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.293042    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7" containerName="revoke-keys"
Nov 03 08:24:03.293100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.293061    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3722fc-983b-4b33-9269-8e559475aff5" containerName="payload-processing"
Nov 03 08:24:03.304599 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.304571    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll"]
Nov 03 08:24:03.304753 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.304689    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll"
Nov 03 08:24:03.310900 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.310875    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:24:03.311090 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.311072    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:24:03.311170 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.311145    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:24:03.311235 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.311192    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:24:03.384265 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.384231    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8db8789-e6e5-490d-8a65-5681e647be89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-616e46-h7xll\" (UID: \"d8db8789-e6e5-490d-8a65-5681e647be89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll"
Nov 03 08:24:03.443368 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.422131    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrl29\" (UniqueName: \"kubernetes.io/projected/d8db8789-e6e5-490d-8a65-5681e647be89-kube-api-access-xrl29\") pod \"maas-api-revoke-keys-e2e-inf-b-616e46-h7xll\" (UID: \"d8db8789-e6e5-490d-8a65-5681e647be89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll"
Nov 03 08:24:03.443934 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.443900    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8db8789-e6e5-490d-8a65-5681e647be89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-616e46-h7xll\" (UID: \"d8db8789-e6e5-490d-8a65-5681e647be89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll"
Nov 03 08:24:03.545590 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.545513    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrl29\" (UniqueName: \"kubernetes.io/projected/d8db8789-e6e5-490d-8a65-5681e647be89-kube-api-access-xrl29\") pod \"maas-api-revoke-keys-e2e-inf-b-616e46-h7xll\" (UID: \"d8db8789-e6e5-490d-8a65-5681e647be89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll"
Nov 03 08:24:03.545751 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.545666    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8db8789-e6e5-490d-8a65-5681e647be89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-616e46-h7xll\" (UID: \"d8db8789-e6e5-490d-8a65-5681e647be89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll"
Nov 03 08:24:03.545751 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.545733    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8db8789-e6e5-490d-8a65-5681e647be89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-616e46-h7xll\" (UID: \"d8db8789-e6e5-490d-8a65-5681e647be89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll"
Nov 03 08:24:03.546367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.546311    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8db8789-e6e5-490d-8a65-5681e647be89-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-616e46-h7xll\" (UID: \"d8db8789-e6e5-490d-8a65-5681e647be89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll"
Nov 03 08:24:03.548463 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.548440    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8db8789-e6e5-490d-8a65-5681e647be89-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-616e46-h7xll\" (UID: \"d8db8789-e6e5-490d-8a65-5681e647be89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll"
Nov 03 08:24:03.552822 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.552748    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrl29\" (UniqueName: \"kubernetes.io/projected/d8db8789-e6e5-490d-8a65-5681e647be89-kube-api-access-xrl29\") pod \"maas-api-revoke-keys-e2e-inf-b-616e46-h7xll\" (UID: \"d8db8789-e6e5-490d-8a65-5681e647be89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll"
Nov 03 08:24:03.625111 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.625086    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll"
Nov 03 08:24:03.826825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.826725    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:24:03.826825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.826806    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv" podUID="198146ed-665b-4801-8396-c980ee6ff4d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:24:03.937091 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:24:03.937057    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8db8789_e6e5_490d_8a65_5681e647be89.slice/crio-07c59b14ebbe9aab824b30e9549c560da1e83b563e189c32974a0f174831b00e WatchSource:0}: Error finding container 07c59b14ebbe9aab824b30e9549c560da1e83b563e189c32974a0f174831b00e: Status 404 returned error can't find the container with id 07c59b14ebbe9aab824b30e9549c560da1e83b563e189c32974a0f174831b00e
Nov 03 08:24:03.937330 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:03.937311    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll"]
Nov 03 08:24:04.027587 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:04.027558    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afa12a2a-bfc8-46f0-9647-1ee6f00447d4" path="/var/lib/kubelet/pods/afa12a2a-bfc8-46f0-9647-1ee6f00447d4/volumes"
Nov 03 08:24:04.131696 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:04.131667    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:24:04.131843 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:04.131726    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:24:04.755612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:04.755568    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll" event={"ID":"d8db8789-e6e5-490d-8a65-5681e647be89","Type":"ContainerStarted","Data":"86bacb13204c4dd5b1b3ac4fd8ef7496baae92694558254546ecffe85da53bb9"}
Nov 03 08:24:04.755894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:04.755623    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll" event={"ID":"d8db8789-e6e5-490d-8a65-5681e647be89","Type":"ContainerStarted","Data":"07c59b14ebbe9aab824b30e9549c560da1e83b563e189c32974a0f174831b00e"}
Nov 03 08:24:04.777634 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:04.777588    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll" podStartSLOduration=1.7775748500000002 podStartE2EDuration="1.77757485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:24:04.774425517 +0000 UTC m=+1842.888889449" watchObservedRunningTime="2026-11-03 08:24:04.77757485 +0000 UTC m=+1842.892038810"
Nov 03 08:24:04.861775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:04.855955    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"]
Nov 03 08:24:04.861775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:04.856193    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy" containerID="cri-o://51d9e728d05eb9ccc97d671ec61308aea3135db9f7491831c67acbf0409a301e" gracePeriod=30
Nov 03 08:24:05.280177 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:05.280138    5070 patch_prober.go:29] interesting pod/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:24:05.280360 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:05.280199    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm" podUID="be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:24:05.766500 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:05.766465    5070 generic.go:356] "Generic (PLEG): container finished" podID="d8db8789-e6e5-490d-8a65-5681e647be89" containerID="86bacb13204c4dd5b1b3ac4fd8ef7496baae92694558254546ecffe85da53bb9" exitCode=0
Nov 03 08:24:05.766694 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:05.766521    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll" event={"ID":"d8db8789-e6e5-490d-8a65-5681e647be89","Type":"ContainerDied","Data":"86bacb13204c4dd5b1b3ac4fd8ef7496baae92694558254546ecffe85da53bb9"}
Nov 03 08:24:05.901421 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:05.901387    5070 patch_prober.go:29] interesting pod/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:24:05.901881 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:05.901448    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:24:07.072202 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:07.071272    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv"]
Nov 03 08:24:07.085708 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:07.084387    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-76ef13-xtsjv"]
Nov 03 08:24:07.341160 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:07.341131    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll"
Nov 03 08:24:07.440481 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:07.440445    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8db8789-e6e5-490d-8a65-5681e647be89-kube-api-access-xrl29\" (UniqueName: \"kubernetes.io/projected/d8db8789-e6e5-490d-8a65-5681e647be89-kube-api-access-xrl29\") pod \"d8db8789-e6e5-490d-8a65-5681e647be89\" (UID: \"d8db8789-e6e5-490d-8a65-5681e647be89\") "
Nov 03 08:24:07.440679 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:07.440562    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8db8789-e6e5-490d-8a65-5681e647be89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8db8789-e6e5-490d-8a65-5681e647be89-maas-api-openshift-ca\") pod \"d8db8789-e6e5-490d-8a65-5681e647be89\" (UID: \"d8db8789-e6e5-490d-8a65-5681e647be89\") "
Nov 03 08:24:07.440679 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:07.440616    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8db8789-e6e5-490d-8a65-5681e647be89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8db8789-e6e5-490d-8a65-5681e647be89-maas-api-xks-ca\") pod \"d8db8789-e6e5-490d-8a65-5681e647be89\" (UID: \"d8db8789-e6e5-490d-8a65-5681e647be89\") "
Nov 03 08:24:07.441023 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:07.440992    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8db8789-e6e5-490d-8a65-5681e647be89-maas-api-openshift-ca" pod "d8db8789-e6e5-490d-8a65-5681e647be89" (UID: "d8db8789-e6e5-490d-8a65-5681e647be89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:07.441440 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:07.441420    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8db8789-e6e5-490d-8a65-5681e647be89-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:07.442635 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:07.442608    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8db8789-e6e5-490d-8a65-5681e647be89-kube-api-access-xrl29" pod "d8db8789-e6e5-490d-8a65-5681e647be89" (UID: "d8db8789-e6e5-490d-8a65-5681e647be89"). InnerVolumeSpecName "kube-api-access-xrl29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:07.442814 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:07.442797    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8db8789-e6e5-490d-8a65-5681e647be89-maas-api-xks-ca" pod "d8db8789-e6e5-490d-8a65-5681e647be89" (UID: "d8db8789-e6e5-490d-8a65-5681e647be89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:24:07.542543 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:07.542509    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrl29\" (UniqueName: \"kubernetes.io/projected/d8db8789-e6e5-490d-8a65-5681e647be89-kube-api-access-xrl29\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:07.542543 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:07.542539    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8db8789-e6e5-490d-8a65-5681e647be89-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:07.786492 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:07.786469    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll"
Nov 03 08:24:07.786623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:07.786556    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll" event={"ID":"d8db8789-e6e5-490d-8a65-5681e647be89","Type":"ContainerDied","Data":"07c59b14ebbe9aab824b30e9549c560da1e83b563e189c32974a0f174831b00e"}
Nov 03 08:24:07.786623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:07.786608    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07c59b14ebbe9aab824b30e9549c560da1e83b563e189c32974a0f174831b00e"
Nov 03 08:24:07.795412 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:07.795384    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod198146ed_665b_4801_8396_c980ee6ff4d4.slice/crio-aaba34aa082b19fa1fe5fd4bf0982a15a6b4e7a9fd5623ca022458171a88a036.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4174a67_8cbf_452f_b382_8efd46377534.slice/crio-a25db031f450eb71ce100592f948cee1701b87545c849891ee22aacf0462587f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe94bdb2_23e9_4f2d_a2ff_32f2703ebaf9.slice/crio-conmon-e4d3100066a6b2b6e4b419aef7f8dd47e9de612dd61cc0a1256e740e23b50e46.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 08:24:08.103689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.100503    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:24:08.119255 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.119230    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f195a785-22bc-46c1-833b-f63a37a2de85" path="/var/lib/kubelet/pods/f195a785-22bc-46c1-833b-f63a37a2de85/volumes"
Nov 03 08:24:08.125964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.125875    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:24:08.207072 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.207035    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-envoy\") pod \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") "
Nov 03 08:24:08.207248 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.207117    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/198146ed-665b-4801-8396-c980ee6ff4d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/198146ed-665b-4801-8396-c980ee6ff4d4-istiod-ca-cert\") pod \"198146ed-665b-4801-8396-c980ee6ff4d4\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") "
Nov 03 08:24:08.207248 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.207148    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/198146ed-665b-4801-8396-c980ee6ff4d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/198146ed-665b-4801-8396-c980ee6ff4d4-istio-podinfo\") pod \"198146ed-665b-4801-8396-c980ee6ff4d4\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") "
Nov 03 08:24:08.207248 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.207222    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-credential-socket\") pod \"198146ed-665b-4801-8396-c980ee6ff4d4\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") "
Nov 03 08:24:08.207475 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.207264    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-data\") pod \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") "
Nov 03 08:24:08.207475 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.207307    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-podinfo\") pod \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") "
Nov 03 08:24:08.209423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.207696    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-credential-socket" pod "198146ed-665b-4801-8396-c980ee6ff4d4" (UID: "198146ed-665b-4801-8396-c980ee6ff4d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.209423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.207804    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-data" pod "be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" (UID: "be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.209423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.207812    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-istio-data\") pod \"198146ed-665b-4801-8396-c980ee6ff4d4\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") "
Nov 03 08:24:08.209423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.207909    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istiod-ca-cert\") pod \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") "
Nov 03 08:24:08.209423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.208003    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-workload-certs\") pod \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") "
Nov 03 08:24:08.209423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.208051    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-kube-api-access-kmmq6\" (UniqueName: \"kubernetes.io/projected/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-kube-api-access-kmmq6\") pod \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") "
Nov 03 08:24:08.209423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.208108    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-credential-socket\") pod \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") "
Nov 03 08:24:08.209423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.208153    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-istio-envoy\") pod \"198146ed-665b-4801-8396-c980ee6ff4d4\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") "
Nov 03 08:24:08.209423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.208274    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-workload-socket\") pod \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") "
Nov 03 08:24:08.209423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.208335    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-workload-certs\") pod \"198146ed-665b-4801-8396-c980ee6ff4d4\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") "
Nov 03 08:24:08.209423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.208376    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-workload-socket\") pod \"198146ed-665b-4801-8396-c980ee6ff4d4\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") "
Nov 03 08:24:08.209423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.208416    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-token\" (UniqueName: \"kubernetes.io/projected/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-token\") pod \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\" (UID: \"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9\") "
Nov 03 08:24:08.209423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.208530    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198146ed-665b-4801-8396-c980ee6ff4d4-istio-token\" (UniqueName: \"kubernetes.io/projected/198146ed-665b-4801-8396-c980ee6ff4d4-istio-token\") pod \"198146ed-665b-4801-8396-c980ee6ff4d4\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") "
Nov 03 08:24:08.209423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.208579    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/198146ed-665b-4801-8396-c980ee6ff4d4-kube-api-access-69lxp\" (UniqueName: \"kubernetes.io/projected/198146ed-665b-4801-8396-c980ee6ff4d4-kube-api-access-69lxp\") pod \"198146ed-665b-4801-8396-c980ee6ff4d4\" (UID: \"198146ed-665b-4801-8396-c980ee6ff4d4\") "
Nov 03 08:24:08.210174 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.209531    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.210174 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.209555    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.211157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.208005    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-istio-data" pod "198146ed-665b-4801-8396-c980ee6ff4d4" (UID: "198146ed-665b-4801-8396-c980ee6ff4d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.211157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.208241    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-workload-certs" pod "be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" (UID: "be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.211157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.208945    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istiod-ca-cert" pod "be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" (UID: "be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:08.211157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.209198    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-workload-certs" pod "198146ed-665b-4801-8396-c980ee6ff4d4" (UID: "198146ed-665b-4801-8396-c980ee6ff4d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.211157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.210392    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-workload-socket" pod "be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" (UID: "be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.211157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.210879    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/198146ed-665b-4801-8396-c980ee6ff4d4-istiod-ca-cert" pod "198146ed-665b-4801-8396-c980ee6ff4d4" (UID: "198146ed-665b-4801-8396-c980ee6ff4d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:08.211713 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.211352    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-credential-socket" pod "be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" (UID: "be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.211713 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.211582    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-workload-socket" pod "198146ed-665b-4801-8396-c980ee6ff4d4" (UID: "198146ed-665b-4801-8396-c980ee6ff4d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.212698 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.212669    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-podinfo" pod "be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" (UID: "be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:08.213935 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.213899    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-kube-api-access-kmmq6" pod "be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" (UID: "be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9"). InnerVolumeSpecName "kube-api-access-kmmq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:08.213935 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.213925    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198146ed-665b-4801-8396-c980ee6ff4d4-kube-api-access-69lxp" pod "198146ed-665b-4801-8396-c980ee6ff4d4" (UID: "198146ed-665b-4801-8396-c980ee6ff4d4"). InnerVolumeSpecName "kube-api-access-69lxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:08.214093 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.213950    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-token" pod "be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" (UID: "be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:08.214234 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.214207    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/198146ed-665b-4801-8396-c980ee6ff4d4-istio-podinfo" pod "198146ed-665b-4801-8396-c980ee6ff4d4" (UID: "198146ed-665b-4801-8396-c980ee6ff4d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:08.214306 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.214236    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-envoy" pod "be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" (UID: "be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.214378 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.214366    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-istio-envoy" pod "198146ed-665b-4801-8396-c980ee6ff4d4" (UID: "198146ed-665b-4801-8396-c980ee6ff4d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.216044 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.216017    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/198146ed-665b-4801-8396-c980ee6ff4d4-istio-token" pod "198146ed-665b-4801-8396-c980ee6ff4d4" (UID: "198146ed-665b-4801-8396-c980ee6ff4d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:08.255519 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.248862    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:24:08.255519 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.252391    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:24:08.311535 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.310950    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4174a67-8cbf-452f-b382-8efd46377534-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4174a67-8cbf-452f-b382-8efd46377534-istio-podinfo\") pod \"d4174a67-8cbf-452f-b382-8efd46377534\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") "
Nov 03 08:24:08.311535 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.310993    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-data\") pod \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") "
Nov 03 08:24:08.311535 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.311028    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-workload-certs\") pod \"d4174a67-8cbf-452f-b382-8efd46377534\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") "
Nov 03 08:24:08.311535 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.311065    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4174a67-8cbf-452f-b382-8efd46377534-kube-api-access-55n5w\" (UniqueName: \"kubernetes.io/projected/d4174a67-8cbf-452f-b382-8efd46377534-kube-api-access-55n5w\") pod \"d4174a67-8cbf-452f-b382-8efd46377534\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") "
Nov 03 08:24:08.311535 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.311140    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-token\" (UniqueName: \"kubernetes.io/projected/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-token\") pod \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") "
Nov 03 08:24:08.311535 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.311198    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-workload-certs\") pod \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") "
Nov 03 08:24:08.311535 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.311261    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-credential-socket\") pod \"d4174a67-8cbf-452f-b382-8efd46377534\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") "
Nov 03 08:24:08.311535 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.311311    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4174a67-8cbf-452f-b382-8efd46377534-istio-token\" (UniqueName: \"kubernetes.io/projected/d4174a67-8cbf-452f-b382-8efd46377534-istio-token\") pod \"d4174a67-8cbf-452f-b382-8efd46377534\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") "
Nov 03 08:24:08.311535 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.311345    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istiod-ca-cert\") pod \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") "
Nov 03 08:24:08.311535 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.311386    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-envoy\") pod \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") "
Nov 03 08:24:08.311535 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.311415    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-podinfo\") pod \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") "
Nov 03 08:24:08.311535 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.311470    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4174a67-8cbf-452f-b382-8efd46377534-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4174a67-8cbf-452f-b382-8efd46377534-istiod-ca-cert\") pod \"d4174a67-8cbf-452f-b382-8efd46377534\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") "
Nov 03 08:24:08.311535 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.311502    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-workload-socket\") pod \"d4174a67-8cbf-452f-b382-8efd46377534\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") "
Nov 03 08:24:08.312563 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.311544    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-kube-api-access-6hc4f\" (UniqueName: \"kubernetes.io/projected/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-kube-api-access-6hc4f\") pod \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") "
Nov 03 08:24:08.312563 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.311587    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-istio-data\") pod \"d4174a67-8cbf-452f-b382-8efd46377534\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") "
Nov 03 08:24:08.312563 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.311662    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-istio-envoy\") pod \"d4174a67-8cbf-452f-b382-8efd46377534\" (UID: \"d4174a67-8cbf-452f-b382-8efd46377534\") "
Nov 03 08:24:08.312563 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.311808    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-workload-socket\") pod \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") "
Nov 03 08:24:08.312563 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.311841    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-credential-socket\") pod \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\" (UID: \"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6\") "
Nov 03 08:24:08.313426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.312964    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.313426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.312989    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.313426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.313003    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.313426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.313019    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.313426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.313032    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmmq6\" (UniqueName: \"kubernetes.io/projected/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-kube-api-access-kmmq6\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.313426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.313045    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.313426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.313058    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.313426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.313070    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.313426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.313084    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.313426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.313098    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/198146ed-665b-4801-8396-c980ee6ff4d4-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.313426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.313112    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.313426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.313124    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/198146ed-665b-4801-8396-c980ee6ff4d4-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.313426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.313138    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69lxp\" (UniqueName: \"kubernetes.io/projected/198146ed-665b-4801-8396-c980ee6ff4d4-kube-api-access-69lxp\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.313426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.313151    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.313426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.313164    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/198146ed-665b-4801-8396-c980ee6ff4d4-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.313426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.313180    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/198146ed-665b-4801-8396-c980ee6ff4d4-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.314461 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.313675    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-credential-socket" pod "b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" (UID: "b8d426d3-a0f7-4485-ab2c-a8e82b12fce6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.314461 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.313880    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istiod-ca-cert" pod "b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" (UID: "b8d426d3-a0f7-4485-ab2c-a8e82b12fce6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:08.314461 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.313997    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-workload-certs" pod "b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" (UID: "b8d426d3-a0f7-4485-ab2c-a8e82b12fce6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.314461 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.314294    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-data" pod "b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" (UID: "b8d426d3-a0f7-4485-ab2c-a8e82b12fce6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.314806 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.314523    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-credential-socket" pod "d4174a67-8cbf-452f-b382-8efd46377534" (UID: "d4174a67-8cbf-452f-b382-8efd46377534"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.315059 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.314936    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-workload-certs" pod "d4174a67-8cbf-452f-b382-8efd46377534" (UID: "d4174a67-8cbf-452f-b382-8efd46377534"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.315264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.315142    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-workload-socket" pod "b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" (UID: "b8d426d3-a0f7-4485-ab2c-a8e82b12fce6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.315485 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.315350    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-istio-data" pod "d4174a67-8cbf-452f-b382-8efd46377534" (UID: "d4174a67-8cbf-452f-b382-8efd46377534"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.316042 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.315717    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4174a67-8cbf-452f-b382-8efd46377534-istiod-ca-cert" pod "d4174a67-8cbf-452f-b382-8efd46377534" (UID: "d4174a67-8cbf-452f-b382-8efd46377534"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:08.316349 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.316257    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-workload-socket" pod "d4174a67-8cbf-452f-b382-8efd46377534" (UID: "d4174a67-8cbf-452f-b382-8efd46377534"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.318355 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.318302    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-envoy" pod "b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" (UID: "b8d426d3-a0f7-4485-ab2c-a8e82b12fce6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.318604 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.318585    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4174a67-8cbf-452f-b382-8efd46377534-istio-podinfo" pod "d4174a67-8cbf-452f-b382-8efd46377534" (UID: "d4174a67-8cbf-452f-b382-8efd46377534"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:08.319369 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.319201    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4174a67-8cbf-452f-b382-8efd46377534-kube-api-access-55n5w" pod "d4174a67-8cbf-452f-b382-8efd46377534" (UID: "d4174a67-8cbf-452f-b382-8efd46377534"). InnerVolumeSpecName "kube-api-access-55n5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:08.319488 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.319462    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-token" pod "b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" (UID: "b8d426d3-a0f7-4485-ab2c-a8e82b12fce6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:08.319812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.319786    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4174a67-8cbf-452f-b382-8efd46377534-istio-token" pod "d4174a67-8cbf-452f-b382-8efd46377534" (UID: "d4174a67-8cbf-452f-b382-8efd46377534"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:08.319920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.319897    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-podinfo" pod "b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" (UID: "b8d426d3-a0f7-4485-ab2c-a8e82b12fce6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:08.319992 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.319967    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-istio-envoy" pod "d4174a67-8cbf-452f-b382-8efd46377534" (UID: "d4174a67-8cbf-452f-b382-8efd46377534"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:08.321804 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.321514    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-kube-api-access-6hc4f" pod "b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" (UID: "b8d426d3-a0f7-4485-ab2c-a8e82b12fce6"). InnerVolumeSpecName "kube-api-access-6hc4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:08.417697 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.417659    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4174a67-8cbf-452f-b382-8efd46377534-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.417697 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.417687    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.417697 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.417699    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.417697 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.417708    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.418119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.417717    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4174a67-8cbf-452f-b382-8efd46377534-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.418119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.417726    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.418119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.417734    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hc4f\" (UniqueName: \"kubernetes.io/projected/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-kube-api-access-6hc4f\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.418119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.417745    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.418119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.417753    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.418119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.417779    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.418119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.417792    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.418119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.417803    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4174a67-8cbf-452f-b382-8efd46377534-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.418119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.417811    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.418119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.417819    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.418119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.417827    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55n5w\" (UniqueName: \"kubernetes.io/projected/d4174a67-8cbf-452f-b382-8efd46377534-kube-api-access-55n5w\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.418119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.417835    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.418119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.417844    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.418119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.417854    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4174a67-8cbf-452f-b382-8efd46377534-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:08.796498 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.796464    5070 generic.go:356] "Generic (PLEG): container finished" podID="be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" containerID="e4d3100066a6b2b6e4b419aef7f8dd47e9de612dd61cc0a1256e740e23b50e46" exitCode=0
Nov 03 08:24:08.796721 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.796528    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"
Nov 03 08:24:08.796721 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.796555    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm" event={"ID":"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9","Type":"ContainerDied","Data":"e4d3100066a6b2b6e4b419aef7f8dd47e9de612dd61cc0a1256e740e23b50e46"}
Nov 03 08:24:08.796721 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.796637    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm" event={"ID":"be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9","Type":"ContainerDied","Data":"9f944b07db8142203ef7ea2fe5beaa7e91d6e1edd87b1c8640dfd837a24998e7"}
Nov 03 08:24:08.796721 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.796672    5070 scope.go:122] "RemoveContainer" containerID="e4d3100066a6b2b6e4b419aef7f8dd47e9de612dd61cc0a1256e740e23b50e46"
Nov 03 08:24:08.799098 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.799065    5070 generic.go:356] "Generic (PLEG): container finished" podID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerID="af8c47943e052cd3b42beaabdc4afa3e932d82b9358e85fdb77720ea38f9921c" exitCode=0
Nov 03 08:24:08.799211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.799098    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" event={"ID":"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6","Type":"ContainerDied","Data":"af8c47943e052cd3b42beaabdc4afa3e932d82b9358e85fdb77720ea38f9921c"}
Nov 03 08:24:08.799211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.799158    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62" event={"ID":"b8d426d3-a0f7-4485-ab2c-a8e82b12fce6","Type":"ContainerDied","Data":"0c20e43ecdb4f45981b80729d7e0bda1ffc98591e3fffd7d713517dbdd4708d6"}
Nov 03 08:24:08.799472 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.799447    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"
Nov 03 08:24:08.800874 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.800853    5070 generic.go:356] "Generic (PLEG): container finished" podID="d4174a67-8cbf-452f-b382-8efd46377534" containerID="a25db031f450eb71ce100592f948cee1701b87545c849891ee22aacf0462587f" exitCode=0
Nov 03 08:24:08.800974 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.800882    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt" event={"ID":"d4174a67-8cbf-452f-b382-8efd46377534","Type":"ContainerDied","Data":"a25db031f450eb71ce100592f948cee1701b87545c849891ee22aacf0462587f"}
Nov 03 08:24:08.800974 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.800922    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt" event={"ID":"d4174a67-8cbf-452f-b382-8efd46377534","Type":"ContainerDied","Data":"8fe9db84b2fe493ba1ea17e540a38cd126bdb7c0cbd5dc5f402c58f6cbc6c9cb"}
Nov 03 08:24:08.800974 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.800932    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"
Nov 03 08:24:08.802584 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.802561    5070 generic.go:356] "Generic (PLEG): container finished" podID="198146ed-665b-4801-8396-c980ee6ff4d4" containerID="aaba34aa082b19fa1fe5fd4bf0982a15a6b4e7a9fd5623ca022458171a88a036" exitCode=0
Nov 03 08:24:08.802733 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.802648    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"
Nov 03 08:24:08.802844 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.802649    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv" event={"ID":"198146ed-665b-4801-8396-c980ee6ff4d4","Type":"ContainerDied","Data":"aaba34aa082b19fa1fe5fd4bf0982a15a6b4e7a9fd5623ca022458171a88a036"}
Nov 03 08:24:08.802844 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.802814    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv" event={"ID":"198146ed-665b-4801-8396-c980ee6ff4d4","Type":"ContainerDied","Data":"261a8a38390427188cd605257dde1373746ce43954264f998c8fbf17b87043c5"}
Nov 03 08:24:08.813460 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.813439    5070 scope.go:122] "RemoveContainer" containerID="e4d3100066a6b2b6e4b419aef7f8dd47e9de612dd61cc0a1256e740e23b50e46"
Nov 03 08:24:08.813741 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:08.813723    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4d3100066a6b2b6e4b419aef7f8dd47e9de612dd61cc0a1256e740e23b50e46\": container with ID starting with e4d3100066a6b2b6e4b419aef7f8dd47e9de612dd61cc0a1256e740e23b50e46 not found: ID does not exist" containerID="e4d3100066a6b2b6e4b419aef7f8dd47e9de612dd61cc0a1256e740e23b50e46"
Nov 03 08:24:08.813867 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.813753    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4d3100066a6b2b6e4b419aef7f8dd47e9de612dd61cc0a1256e740e23b50e46"} err="failed to get container status \"e4d3100066a6b2b6e4b419aef7f8dd47e9de612dd61cc0a1256e740e23b50e46\": rpc error: code = NotFound desc = could not find container \"e4d3100066a6b2b6e4b419aef7f8dd47e9de612dd61cc0a1256e740e23b50e46\": container with ID starting with e4d3100066a6b2b6e4b419aef7f8dd47e9de612dd61cc0a1256e740e23b50e46 not found: ID does not exist"
Nov 03 08:24:08.813867 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.813789    5070 scope.go:122] "RemoveContainer" containerID="af8c47943e052cd3b42beaabdc4afa3e932d82b9358e85fdb77720ea38f9921c"
Nov 03 08:24:08.828855 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.828743    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"]
Nov 03 08:24:08.831001 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.830980    5070 scope.go:122] "RemoveContainer" containerID="5c89f506e79ac82b260040c7148b1e731c64138a5485a246e28a4ad6a3fb01d1"
Nov 03 08:24:08.834636 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.834618    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-q46zm"]
Nov 03 08:24:08.842892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.842869    5070 scope.go:122] "RemoveContainer" containerID="af8c47943e052cd3b42beaabdc4afa3e932d82b9358e85fdb77720ea38f9921c"
Nov 03 08:24:08.843161 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:08.843133    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af8c47943e052cd3b42beaabdc4afa3e932d82b9358e85fdb77720ea38f9921c\": container with ID starting with af8c47943e052cd3b42beaabdc4afa3e932d82b9358e85fdb77720ea38f9921c not found: ID does not exist" containerID="af8c47943e052cd3b42beaabdc4afa3e932d82b9358e85fdb77720ea38f9921c"
Nov 03 08:24:08.843228 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.843172    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af8c47943e052cd3b42beaabdc4afa3e932d82b9358e85fdb77720ea38f9921c"} err="failed to get container status \"af8c47943e052cd3b42beaabdc4afa3e932d82b9358e85fdb77720ea38f9921c\": rpc error: code = NotFound desc = could not find container \"af8c47943e052cd3b42beaabdc4afa3e932d82b9358e85fdb77720ea38f9921c\": container with ID starting with af8c47943e052cd3b42beaabdc4afa3e932d82b9358e85fdb77720ea38f9921c not found: ID does not exist"
Nov 03 08:24:08.843228 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.843196    5070 scope.go:122] "RemoveContainer" containerID="5c89f506e79ac82b260040c7148b1e731c64138a5485a246e28a4ad6a3fb01d1"
Nov 03 08:24:08.843421 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:08.843401    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c89f506e79ac82b260040c7148b1e731c64138a5485a246e28a4ad6a3fb01d1\": container with ID starting with 5c89f506e79ac82b260040c7148b1e731c64138a5485a246e28a4ad6a3fb01d1 not found: ID does not exist" containerID="5c89f506e79ac82b260040c7148b1e731c64138a5485a246e28a4ad6a3fb01d1"
Nov 03 08:24:08.843460 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.843427    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c89f506e79ac82b260040c7148b1e731c64138a5485a246e28a4ad6a3fb01d1"} err="failed to get container status \"5c89f506e79ac82b260040c7148b1e731c64138a5485a246e28a4ad6a3fb01d1\": rpc error: code = NotFound desc = could not find container \"5c89f506e79ac82b260040c7148b1e731c64138a5485a246e28a4ad6a3fb01d1\": container with ID starting with 5c89f506e79ac82b260040c7148b1e731c64138a5485a246e28a4ad6a3fb01d1 not found: ID does not exist"
Nov 03 08:24:08.843460 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.843446    5070 scope.go:122] "RemoveContainer" containerID="a25db031f450eb71ce100592f948cee1701b87545c849891ee22aacf0462587f"
Nov 03 08:24:08.846292 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.846267    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"]
Nov 03 08:24:08.851126 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.851106    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-lmqpv"]
Nov 03 08:24:08.857433 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.857412    5070 scope.go:122] "RemoveContainer" containerID="a25db031f450eb71ce100592f948cee1701b87545c849891ee22aacf0462587f"
Nov 03 08:24:08.857873 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:08.857852    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a25db031f450eb71ce100592f948cee1701b87545c849891ee22aacf0462587f\": container with ID starting with a25db031f450eb71ce100592f948cee1701b87545c849891ee22aacf0462587f not found: ID does not exist" containerID="a25db031f450eb71ce100592f948cee1701b87545c849891ee22aacf0462587f"
Nov 03 08:24:08.857975 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.857877    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25db031f450eb71ce100592f948cee1701b87545c849891ee22aacf0462587f"} err="failed to get container status \"a25db031f450eb71ce100592f948cee1701b87545c849891ee22aacf0462587f\": rpc error: code = NotFound desc = could not find container \"a25db031f450eb71ce100592f948cee1701b87545c849891ee22aacf0462587f\": container with ID starting with a25db031f450eb71ce100592f948cee1701b87545c849891ee22aacf0462587f not found: ID does not exist"
Nov 03 08:24:08.857975 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.857893    5070 scope.go:122] "RemoveContainer" containerID="aaba34aa082b19fa1fe5fd4bf0982a15a6b4e7a9fd5623ca022458171a88a036"
Nov 03 08:24:08.863984 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.862334    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"]
Nov 03 08:24:08.868552 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.868528    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-jxwrt"]
Nov 03 08:24:08.920406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.911032    5070 scope.go:122] "RemoveContainer" containerID="aaba34aa082b19fa1fe5fd4bf0982a15a6b4e7a9fd5623ca022458171a88a036"
Nov 03 08:24:08.920406 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:08.920232    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaba34aa082b19fa1fe5fd4bf0982a15a6b4e7a9fd5623ca022458171a88a036\": container with ID starting with aaba34aa082b19fa1fe5fd4bf0982a15a6b4e7a9fd5623ca022458171a88a036 not found: ID does not exist" containerID="aaba34aa082b19fa1fe5fd4bf0982a15a6b4e7a9fd5623ca022458171a88a036"
Nov 03 08:24:08.920406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.920277    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaba34aa082b19fa1fe5fd4bf0982a15a6b4e7a9fd5623ca022458171a88a036"} err="failed to get container status \"aaba34aa082b19fa1fe5fd4bf0982a15a6b4e7a9fd5623ca022458171a88a036\": rpc error: code = NotFound desc = could not find container \"aaba34aa082b19fa1fe5fd4bf0982a15a6b4e7a9fd5623ca022458171a88a036\": container with ID starting with aaba34aa082b19fa1fe5fd4bf0982a15a6b4e7a9fd5623ca022458171a88a036 not found: ID does not exist"
Nov 03 08:24:08.983599 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.978627    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"]
Nov 03 08:24:09.007782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:08.997073    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e0f576-openshift-default-66fd54fdf7-snf62"]
Nov 03 08:24:10.032835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.030631    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="198146ed-665b-4801-8396-c980ee6ff4d4" path="/var/lib/kubelet/pods/198146ed-665b-4801-8396-c980ee6ff4d4/volumes"
Nov 03 08:24:10.032835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.031646    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" path="/var/lib/kubelet/pods/b8d426d3-a0f7-4485-ab2c-a8e82b12fce6/volumes"
Nov 03 08:24:10.032835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.032363    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" path="/var/lib/kubelet/pods/be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9/volumes"
Nov 03 08:24:10.033494 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.033473    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4174a67-8cbf-452f-b382-8efd46377534" path="/var/lib/kubelet/pods/d4174a67-8cbf-452f-b382-8efd46377534/volumes"
Nov 03 08:24:10.075376 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.075350    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"]
Nov 03 08:24:10.075811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.075747    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss" podUID="9de4f8ca-60d4-443b-9307-e7c763130938" containerName="istio-proxy" containerID="cri-o://70c1b4eae07e31e0c089d4f896026fc73ce3527a0c3fd1ea6370da4d5460c15d" gracePeriod=30
Nov 03 08:24:10.080526 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.080340    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"]
Nov 03 08:24:10.080647 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.080577    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy" containerID="cri-o://11a078b3ccf228e563e1e3b67d99f29a2327698ee3cc23f7dc65a574eccf3f83" gracePeriod=30
Nov 03 08:24:10.087779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.086988    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"]
Nov 03 08:24:10.087779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.087158    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r" podUID="bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" containerName="istio-proxy" containerID="cri-o://a967eb664dd1a02bc3f7caa6e51c62911ebadfe4e74a25d5e7a237a847450b90" gracePeriod=30
Nov 03 08:24:10.093428 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.093410    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"]
Nov 03 08:24:10.093627 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.093603    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8" podUID="5edae0d0-8211-41ba-8b4b-154fa2b3f840" containerName="istio-proxy" containerID="cri-o://65447f53b80c7570e545e62a80ac6de929e011ddf6aca53a6715a7d0dbbcf33c" gracePeriod=30
Nov 03 08:24:10.100544 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.100525    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"]
Nov 03 08:24:10.100814 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.100792    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk" podUID="212c33bc-85a6-4ff5-a96a-eda0064fab63" containerName="istio-proxy" containerID="cri-o://906ca3f54a6077a62a9475c973ab73f82b8058b800fa080208b163d7c16361e5" gracePeriod=30
Nov 03 08:24:10.430643 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.430607    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:24:10.502215 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.502181    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istiod-ca-cert\") pod \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") "
Nov 03 08:24:10.502388 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.502255    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-credential-socket\") pod \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") "
Nov 03 08:24:10.502388 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.502355    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-data\") pod \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") "
Nov 03 08:24:10.502548 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.502458    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-envoy\") pod \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") "
Nov 03 08:24:10.502548 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.502497    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-podinfo\") pod \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") "
Nov 03 08:24:10.502548 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.502538    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-credential-socket" pod "df4cc63e-36c7-4e6e-87ab-64dea6237e42" (UID: "df4cc63e-36c7-4e6e-87ab-64dea6237e42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:10.502719 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.502568    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-token\" (UniqueName: \"kubernetes.io/projected/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-token\") pod \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") "
Nov 03 08:24:10.502719 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.502624    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-workload-certs\") pod \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") "
Nov 03 08:24:10.502719 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.502665    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istiod-ca-cert" pod "df4cc63e-36c7-4e6e-87ab-64dea6237e42" (UID: "df4cc63e-36c7-4e6e-87ab-64dea6237e42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:10.502719 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.502669    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-data" pod "df4cc63e-36c7-4e6e-87ab-64dea6237e42" (UID: "df4cc63e-36c7-4e6e-87ab-64dea6237e42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:10.502999 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.502822    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df4cc63e-36c7-4e6e-87ab-64dea6237e42-kube-api-access-s4tx5\" (UniqueName: \"kubernetes.io/projected/df4cc63e-36c7-4e6e-87ab-64dea6237e42-kube-api-access-s4tx5\") pod \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") "
Nov 03 08:24:10.502999 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.502907    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-workload-socket\") pod \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\" (UID: \"df4cc63e-36c7-4e6e-87ab-64dea6237e42\") "
Nov 03 08:24:10.503226 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.503200    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-workload-socket" pod "df4cc63e-36c7-4e6e-87ab-64dea6237e42" (UID: "df4cc63e-36c7-4e6e-87ab-64dea6237e42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:10.503435 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.503412    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-workload-certs" pod "df4cc63e-36c7-4e6e-87ab-64dea6237e42" (UID: "df4cc63e-36c7-4e6e-87ab-64dea6237e42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:10.503862 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.503836    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.503962 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.503866    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.503962 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.503881    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.503962 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.503895    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.503962 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.503912    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.505455 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.505427    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-envoy" pod "df4cc63e-36c7-4e6e-87ab-64dea6237e42" (UID: "df4cc63e-36c7-4e6e-87ab-64dea6237e42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:10.505455 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.505431    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-token" pod "df4cc63e-36c7-4e6e-87ab-64dea6237e42" (UID: "df4cc63e-36c7-4e6e-87ab-64dea6237e42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:10.505592 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.505545    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4cc63e-36c7-4e6e-87ab-64dea6237e42-kube-api-access-s4tx5" pod "df4cc63e-36c7-4e6e-87ab-64dea6237e42" (UID: "df4cc63e-36c7-4e6e-87ab-64dea6237e42"). InnerVolumeSpecName "kube-api-access-s4tx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:10.506035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.506010    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-podinfo" pod "df4cc63e-36c7-4e6e-87ab-64dea6237e42" (UID: "df4cc63e-36c7-4e6e-87ab-64dea6237e42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:10.605471 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.605418    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.605471 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.605442    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4tx5\" (UniqueName: \"kubernetes.io/projected/df4cc63e-36c7-4e6e-87ab-64dea6237e42-kube-api-access-s4tx5\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.605471 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.605454    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.605471 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.605463    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df4cc63e-36c7-4e6e-87ab-64dea6237e42-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:10.742903 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.742874    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p"]
Nov 03 08:24:10.743594 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.743578    5070 cpu_manager.go:414] "containerMap: removing container" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy"
Nov 03 08:24:10.743639 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.743595    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy"
Nov 03 08:24:10.743639 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.743614    5070 cpu_manager.go:414] "containerMap: removing container" podUID="d8db8789-e6e5-490d-8a65-5681e647be89" containerName="revoke-keys"
Nov 03 08:24:10.743639 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.743619    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8db8789-e6e5-490d-8a65-5681e647be89" containerName="revoke-keys"
Nov 03 08:24:10.743639 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.743637    5070 cpu_manager.go:414] "containerMap: removing container" podUID="d4174a67-8cbf-452f-b382-8efd46377534" containerName="istio-proxy"
Nov 03 08:24:10.743839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.743642    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4174a67-8cbf-452f-b382-8efd46377534" containerName="istio-proxy"
Nov 03 08:24:10.743839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.743656    5070 cpu_manager.go:414] "containerMap: removing container" podUID="be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" containerName="istio-proxy"
Nov 03 08:24:10.743839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.743661    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" containerName="istio-proxy"
Nov 03 08:24:10.743839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.743667    5070 cpu_manager.go:414] "containerMap: removing container" podUID="198146ed-665b-4801-8396-c980ee6ff4d4" containerName="istio-proxy"
Nov 03 08:24:10.743839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.743672    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="198146ed-665b-4801-8396-c980ee6ff4d4" containerName="istio-proxy"
Nov 03 08:24:10.743839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.743688    5070 cpu_manager.go:414] "containerMap: removing container" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy"
Nov 03 08:24:10.743839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.743692    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy"
Nov 03 08:24:10.743839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.743703    5070 cpu_manager.go:414] "containerMap: removing container" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy"
Nov 03 08:24:10.743839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.743708    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy"
Nov 03 08:24:10.743839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.743816    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4174a67-8cbf-452f-b382-8efd46377534" containerName="istio-proxy"
Nov 03 08:24:10.743839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.743824    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="be94bdb2-23e9-4f2d-a2ff-32f2703ebaf9" containerName="istio-proxy"
Nov 03 08:24:10.743839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.743832    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="198146ed-665b-4801-8396-c980ee6ff4d4" containerName="istio-proxy"
Nov 03 08:24:10.743839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.743841    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerName="istio-proxy"
Nov 03 08:24:10.744232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.743850    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy"
Nov 03 08:24:10.744232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.743861    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d426d3-a0f7-4485-ab2c-a8e82b12fce6" containerName="istio-proxy"
Nov 03 08:24:10.744232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.743869    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8db8789-e6e5-490d-8a65-5681e647be89" containerName="revoke-keys"
Nov 03 08:24:10.747896 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.747877    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p"
Nov 03 08:24:10.751926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.750623    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:24:10.751926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.750712    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:24:10.751926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.750824    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:24:10.751926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.750902    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:24:10.755844 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.755814    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p"]
Nov 03 08:24:10.808839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.808799    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsq4h\" (UniqueName: \"kubernetes.io/projected/803348c9-5802-4b5a-836d-462fb6ebc404-kube-api-access-nsq4h\") pod \"maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p\" (UID: \"803348c9-5802-4b5a-836d-462fb6ebc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p"
Nov 03 08:24:10.809003 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.808986    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/803348c9-5802-4b5a-836d-462fb6ebc404-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p\" (UID: \"803348c9-5802-4b5a-836d-462fb6ebc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p"
Nov 03 08:24:10.809073 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.809040    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/803348c9-5802-4b5a-836d-462fb6ebc404-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p\" (UID: \"803348c9-5802-4b5a-836d-462fb6ebc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p"
Nov 03 08:24:10.898241 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.898171    5070 generic.go:356] "Generic (PLEG): container finished" podID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" containerID="51d9e728d05eb9ccc97d671ec61308aea3135db9f7491831c67acbf0409a301e" exitCode=0
Nov 03 08:24:10.898241 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.898220    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"
Nov 03 08:24:10.898405 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.898247    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" event={"ID":"df4cc63e-36c7-4e6e-87ab-64dea6237e42","Type":"ContainerDied","Data":"51d9e728d05eb9ccc97d671ec61308aea3135db9f7491831c67acbf0409a301e"}
Nov 03 08:24:10.898405 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.898299    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8" event={"ID":"df4cc63e-36c7-4e6e-87ab-64dea6237e42","Type":"ContainerDied","Data":"dbd37e560bad7f669186c031613fcda1c37396481ba191eceedc3f43534af13a"}
Nov 03 08:24:10.898405 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.898323    5070 scope.go:122] "RemoveContainer" containerID="51d9e728d05eb9ccc97d671ec61308aea3135db9f7491831c67acbf0409a301e"
Nov 03 08:24:10.910956 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.910925    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsq4h\" (UniqueName: \"kubernetes.io/projected/803348c9-5802-4b5a-836d-462fb6ebc404-kube-api-access-nsq4h\") pod \"maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p\" (UID: \"803348c9-5802-4b5a-836d-462fb6ebc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p"
Nov 03 08:24:10.911042 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.910982    5070 scope.go:122] "RemoveContainer" containerID="51d9e728d05eb9ccc97d671ec61308aea3135db9f7491831c67acbf0409a301e"
Nov 03 08:24:10.911098 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.911055    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/803348c9-5802-4b5a-836d-462fb6ebc404-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p\" (UID: \"803348c9-5802-4b5a-836d-462fb6ebc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p"
Nov 03 08:24:10.911162 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.911110    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/803348c9-5802-4b5a-836d-462fb6ebc404-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p\" (UID: \"803348c9-5802-4b5a-836d-462fb6ebc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p"
Nov 03 08:24:10.911281 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:10.911261    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51d9e728d05eb9ccc97d671ec61308aea3135db9f7491831c67acbf0409a301e\": container with ID starting with 51d9e728d05eb9ccc97d671ec61308aea3135db9f7491831c67acbf0409a301e not found: ID does not exist" containerID="51d9e728d05eb9ccc97d671ec61308aea3135db9f7491831c67acbf0409a301e"
Nov 03 08:24:10.911346 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.911286    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51d9e728d05eb9ccc97d671ec61308aea3135db9f7491831c67acbf0409a301e"} err="failed to get container status \"51d9e728d05eb9ccc97d671ec61308aea3135db9f7491831c67acbf0409a301e\": rpc error: code = NotFound desc = could not find container \"51d9e728d05eb9ccc97d671ec61308aea3135db9f7491831c67acbf0409a301e\": container with ID starting with 51d9e728d05eb9ccc97d671ec61308aea3135db9f7491831c67acbf0409a301e not found: ID does not exist"
Nov 03 08:24:10.911858 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.911836    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/803348c9-5802-4b5a-836d-462fb6ebc404-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p\" (UID: \"803348c9-5802-4b5a-836d-462fb6ebc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p"
Nov 03 08:24:10.913485 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.913460    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/803348c9-5802-4b5a-836d-462fb6ebc404-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p\" (UID: \"803348c9-5802-4b5a-836d-462fb6ebc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p"
Nov 03 08:24:10.919327 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:10.919306    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsq4h\" (UniqueName: \"kubernetes.io/projected/803348c9-5802-4b5a-836d-462fb6ebc404-kube-api-access-nsq4h\") pod \"maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p\" (UID: \"803348c9-5802-4b5a-836d-462fb6ebc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p"
Nov 03 08:24:11.011649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:11.011610    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"]
Nov 03 08:24:11.018359 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:11.018336    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-15a633-openshift-default-7897f7cd7f-454d8"]
Nov 03 08:24:11.073382 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:11.073352    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p"
Nov 03 08:24:11.358533 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:11.358506    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p"]
Nov 03 08:24:11.359423 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:24:11.359384    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod803348c9_5802_4b5a_836d_462fb6ebc404.slice/crio-95c719af794091b6caf9c79c3e6d02b85a1001fe2de35dd9a492ddaf5d60481b WatchSource:0}: Error finding container 95c719af794091b6caf9c79c3e6d02b85a1001fe2de35dd9a492ddaf5d60481b: Status 404 returned error can't find the container with id 95c719af794091b6caf9c79c3e6d02b85a1001fe2de35dd9a492ddaf5d60481b
Nov 03 08:24:11.908984 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:11.908945    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p" event={"ID":"803348c9-5802-4b5a-836d-462fb6ebc404","Type":"ContainerStarted","Data":"8456e11dd25c095191c6ad4110b287f538a7ebac3e76d00be6ea9786a7c3f70e"}
Nov 03 08:24:11.909500 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:11.909003    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p" event={"ID":"803348c9-5802-4b5a-836d-462fb6ebc404","Type":"ContainerStarted","Data":"95c719af794091b6caf9c79c3e6d02b85a1001fe2de35dd9a492ddaf5d60481b"}
Nov 03 08:24:11.926233 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:11.926151    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p" podStartSLOduration=1.9261383410000001 podStartE2EDuration="1.926138341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:24:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:24:11.923204894 +0000 UTC m=+1850.037668824" watchObservedRunningTime="2026-11-03 08:24:11.926138341 +0000 UTC m=+1850.040602273"
Nov 03 08:24:12.031656 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:12.031624    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df4cc63e-36c7-4e6e-87ab-64dea6237e42" path="/var/lib/kubelet/pods/df4cc63e-36c7-4e6e-87ab-64dea6237e42/volumes"
Nov 03 08:24:12.922132 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:12.922101    5070 generic.go:356] "Generic (PLEG): container finished" podID="803348c9-5802-4b5a-836d-462fb6ebc404" containerID="8456e11dd25c095191c6ad4110b287f538a7ebac3e76d00be6ea9786a7c3f70e" exitCode=0
Nov 03 08:24:12.922570 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:12.922191    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p" event={"ID":"803348c9-5802-4b5a-836d-462fb6ebc404","Type":"ContainerDied","Data":"8456e11dd25c095191c6ad4110b287f538a7ebac3e76d00be6ea9786a7c3f70e"}
Nov 03 08:24:13.687186 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:13.687146    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"]
Nov 03 08:24:13.687399 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:13.687369    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx" podUID="d7faa6ae-27c1-4a98-8c83-4a3fb1669835" containerName="istio-proxy" containerID="cri-o://020923369facd8800e378661dd8b7b52e1428e0e7aa851b1044a81716cc139f8" gracePeriod=30
Nov 03 08:24:14.301840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:14.301817    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p"
Nov 03 08:24:14.479245 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:14.479218    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/803348c9-5802-4b5a-836d-462fb6ebc404-kube-api-access-nsq4h\" (UniqueName: \"kubernetes.io/projected/803348c9-5802-4b5a-836d-462fb6ebc404-kube-api-access-nsq4h\") pod \"803348c9-5802-4b5a-836d-462fb6ebc404\" (UID: \"803348c9-5802-4b5a-836d-462fb6ebc404\") "
Nov 03 08:24:14.479447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:14.479326    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/803348c9-5802-4b5a-836d-462fb6ebc404-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/803348c9-5802-4b5a-836d-462fb6ebc404-maas-api-openshift-ca\") pod \"803348c9-5802-4b5a-836d-462fb6ebc404\" (UID: \"803348c9-5802-4b5a-836d-462fb6ebc404\") "
Nov 03 08:24:14.479592 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:14.479485    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/803348c9-5802-4b5a-836d-462fb6ebc404-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/803348c9-5802-4b5a-836d-462fb6ebc404-maas-api-xks-ca\") pod \"803348c9-5802-4b5a-836d-462fb6ebc404\" (UID: \"803348c9-5802-4b5a-836d-462fb6ebc404\") "
Nov 03 08:24:14.479709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:14.479683    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/803348c9-5802-4b5a-836d-462fb6ebc404-maas-api-openshift-ca" pod "803348c9-5802-4b5a-836d-462fb6ebc404" (UID: "803348c9-5802-4b5a-836d-462fb6ebc404"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:14.480152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:14.480131    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/803348c9-5802-4b5a-836d-462fb6ebc404-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:14.481544 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:14.481518    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803348c9-5802-4b5a-836d-462fb6ebc404-kube-api-access-nsq4h" pod "803348c9-5802-4b5a-836d-462fb6ebc404" (UID: "803348c9-5802-4b5a-836d-462fb6ebc404"). InnerVolumeSpecName "kube-api-access-nsq4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:14.481916 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:14.481890    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/803348c9-5802-4b5a-836d-462fb6ebc404-maas-api-xks-ca" pod "803348c9-5802-4b5a-836d-462fb6ebc404" (UID: "803348c9-5802-4b5a-836d-462fb6ebc404"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:24:14.583851 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:14.583806    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsq4h\" (UniqueName: \"kubernetes.io/projected/803348c9-5802-4b5a-836d-462fb6ebc404-kube-api-access-nsq4h\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:14.584063 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:14.583857    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/803348c9-5802-4b5a-836d-462fb6ebc404-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:14.991747 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:14.991719    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p"
Nov 03 08:24:14.991933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:14.991753    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p" event={"ID":"803348c9-5802-4b5a-836d-462fb6ebc404","Type":"ContainerDied","Data":"95c719af794091b6caf9c79c3e6d02b85a1001fe2de35dd9a492ddaf5d60481b"}
Nov 03 08:24:14.991933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:14.991832    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95c719af794091b6caf9c79c3e6d02b85a1001fe2de35dd9a492ddaf5d60481b"
Nov 03 08:24:15.616684 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.616651    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll"]
Nov 03 08:24:15.627774 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.627729    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-616e46-h7xll"]
Nov 03 08:24:15.734632 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.734601    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:24:15.918807 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.918769    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-workload-socket\") pod \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") "
Nov 03 08:24:15.918924 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.918900    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istiod-ca-cert\") pod \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") "
Nov 03 08:24:15.919103 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.918959    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-credential-socket\") pod \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") "
Nov 03 08:24:15.919103 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.918993    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-envoy\") pod \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") "
Nov 03 08:24:15.919103 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.919011    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-workload-socket" pod "bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" (UID: "bb569224-4a9c-4da1-b5c4-cb6ea74b68bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:15.919103 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.919025    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-data\") pod \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") "
Nov 03 08:24:15.919103 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.919102    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-podinfo\") pod \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") "
Nov 03 08:24:15.919402 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.919186    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-kube-api-access-tckst\" (UniqueName: \"kubernetes.io/projected/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-kube-api-access-tckst\") pod \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") "
Nov 03 08:24:15.919402 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.919283    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-token\" (UniqueName: \"kubernetes.io/projected/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-token\") pod \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") "
Nov 03 08:24:15.919402 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.919397    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-workload-certs\") pod \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\" (UID: \"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd\") "
Nov 03 08:24:15.919570 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.919393    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-data" pod "bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" (UID: "bb569224-4a9c-4da1-b5c4-cb6ea74b68bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:15.919570 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.919414    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istiod-ca-cert" pod "bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" (UID: "bb569224-4a9c-4da1-b5c4-cb6ea74b68bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:15.920180 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.919717    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-credential-socket" pod "bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" (UID: "bb569224-4a9c-4da1-b5c4-cb6ea74b68bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:15.920180 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.920107    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-workload-certs" pod "bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" (UID: "bb569224-4a9c-4da1-b5c4-cb6ea74b68bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:15.920180 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.920127    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:15.920180 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.920145    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:15.920180 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.920154    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:15.920180 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.920162    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:15.921817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.921750    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-podinfo" pod "bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" (UID: "bb569224-4a9c-4da1-b5c4-cb6ea74b68bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:15.927024 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.927000    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-envoy" pod "bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" (UID: "bb569224-4a9c-4da1-b5c4-cb6ea74b68bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:15.927115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.927022    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-token" pod "bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" (UID: "bb569224-4a9c-4da1-b5c4-cb6ea74b68bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:15.928347 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.928325    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-kube-api-access-tckst" pod "bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" (UID: "bb569224-4a9c-4da1-b5c4-cb6ea74b68bd"). InnerVolumeSpecName "kube-api-access-tckst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:15.963101 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.963081    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:24:15.965196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.965179    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:24:15.969225 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.969208    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:24:15.972997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:15.972979    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:24:16.004472 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.004439    5070 generic.go:356] "Generic (PLEG): container finished" podID="9de4f8ca-60d4-443b-9307-e7c763130938" containerID="70c1b4eae07e31e0c089d4f896026fc73ce3527a0c3fd1ea6370da4d5460c15d" exitCode=0
Nov 03 08:24:16.004635 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.004494    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"
Nov 03 08:24:16.004635 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.004524    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss" event={"ID":"9de4f8ca-60d4-443b-9307-e7c763130938","Type":"ContainerDied","Data":"70c1b4eae07e31e0c089d4f896026fc73ce3527a0c3fd1ea6370da4d5460c15d"}
Nov 03 08:24:16.004635 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.004605    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss" event={"ID":"9de4f8ca-60d4-443b-9307-e7c763130938","Type":"ContainerDied","Data":"bca339d40c2c3acdff46691d2831c4a29ad7e0fdf8b396d6b3f920548986b755"}
Nov 03 08:24:16.004892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.004642    5070 scope.go:122] "RemoveContainer" containerID="70c1b4eae07e31e0c089d4f896026fc73ce3527a0c3fd1ea6370da4d5460c15d"
Nov 03 08:24:16.007086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.007059    5070 generic.go:356] "Generic (PLEG): container finished" podID="5edae0d0-8211-41ba-8b4b-154fa2b3f840" containerID="65447f53b80c7570e545e62a80ac6de929e011ddf6aca53a6715a7d0dbbcf33c" exitCode=0
Nov 03 08:24:16.007214 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.007194    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8" event={"ID":"5edae0d0-8211-41ba-8b4b-154fa2b3f840","Type":"ContainerDied","Data":"65447f53b80c7570e545e62a80ac6de929e011ddf6aca53a6715a7d0dbbcf33c"}
Nov 03 08:24:16.007282 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.007242    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8" event={"ID":"5edae0d0-8211-41ba-8b4b-154fa2b3f840","Type":"ContainerDied","Data":"a5b91813b299ec77839285ac83314cf0609bc182255f99609dc020a06c874e14"}
Nov 03 08:24:16.009157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.007341    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"
Nov 03 08:24:16.009157 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.009103    5070 generic.go:356] "Generic (PLEG): container finished" podID="212c33bc-85a6-4ff5-a96a-eda0064fab63" containerID="906ca3f54a6077a62a9475c973ab73f82b8058b800fa080208b163d7c16361e5" exitCode=0
Nov 03 08:24:16.009274 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.009254    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk" event={"ID":"212c33bc-85a6-4ff5-a96a-eda0064fab63","Type":"ContainerDied","Data":"906ca3f54a6077a62a9475c973ab73f82b8058b800fa080208b163d7c16361e5"}
Nov 03 08:24:16.009333 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.009300    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk" event={"ID":"212c33bc-85a6-4ff5-a96a-eda0064fab63","Type":"ContainerDied","Data":"30ee49bd3e0fd4d1b007c0abb82587e4ef0ca53e9f8cbbc69d603aa08e0d4c71"}
Nov 03 08:24:16.009644 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.009368    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"
Nov 03 08:24:16.011447 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.011424    5070 generic.go:356] "Generic (PLEG): container finished" podID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerID="11a078b3ccf228e563e1e3b67d99f29a2327698ee3cc23f7dc65a574eccf3f83" exitCode=0
Nov 03 08:24:16.011621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.011599    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"
Nov 03 08:24:16.011933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.011909    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" event={"ID":"acaf418a-affb-4714-a5ce-5673ed5be68e","Type":"ContainerDied","Data":"11a078b3ccf228e563e1e3b67d99f29a2327698ee3cc23f7dc65a574eccf3f83"}
Nov 03 08:24:16.012023 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.011957    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j" event={"ID":"acaf418a-affb-4714-a5ce-5673ed5be68e","Type":"ContainerDied","Data":"6a9ce820723071620e4fdef5154b561b53e3e39b3679dcc7badbca8eb5481fd8"}
Nov 03 08:24:16.013668 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.013646    5070 generic.go:356] "Generic (PLEG): container finished" podID="bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" containerID="a967eb664dd1a02bc3f7caa6e51c62911ebadfe4e74a25d5e7a237a847450b90" exitCode=0
Nov 03 08:24:16.013862 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.013689    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"
Nov 03 08:24:16.014040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.013706    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r" event={"ID":"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd","Type":"ContainerDied","Data":"a967eb664dd1a02bc3f7caa6e51c62911ebadfe4e74a25d5e7a237a847450b90"}
Nov 03 08:24:16.014140 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.014059    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r" event={"ID":"bb569224-4a9c-4da1-b5c4-cb6ea74b68bd","Type":"ContainerDied","Data":"b0d027fcb178d8878e9d20641066ae94ec004a5866fd579a6f22a1c82646221b"}
Nov 03 08:24:16.020824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.020799    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-podinfo\") pod \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.020837    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-token\" (UniqueName: \"kubernetes.io/projected/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-token\") pod \"212c33bc-85a6-4ff5-a96a-eda0064fab63\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.020863    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-workload-certs\") pod \"212c33bc-85a6-4ff5-a96a-eda0064fab63\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.020894    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9de4f8ca-60d4-443b-9307-e7c763130938-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de4f8ca-60d4-443b-9307-e7c763130938-istiod-ca-cert\") pod \"9de4f8ca-60d4-443b-9307-e7c763130938\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.020944    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-credential-socket\") pod \"acaf418a-affb-4714-a5ce-5673ed5be68e\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.020972    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de4f8ca-60d4-443b-9307-e7c763130938-istio-token\" (UniqueName: \"kubernetes.io/projected/9de4f8ca-60d4-443b-9307-e7c763130938-istio-token\") pod \"9de4f8ca-60d4-443b-9307-e7c763130938\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.021011    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-workload-socket\") pod \"212c33bc-85a6-4ff5-a96a-eda0064fab63\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.021030    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-istio-data\") pod \"9de4f8ca-60d4-443b-9307-e7c763130938\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.021067    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acaf418a-affb-4714-a5ce-5673ed5be68e-kube-api-access-lnkh2\" (UniqueName: \"kubernetes.io/projected/acaf418a-affb-4714-a5ce-5673ed5be68e-kube-api-access-lnkh2\") pod \"acaf418a-affb-4714-a5ce-5673ed5be68e\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.021090    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-workload-certs\") pod \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.021123    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istiod-ca-cert\") pod \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.021145    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5edae0d0-8211-41ba-8b4b-154fa2b3f840-kube-api-access-bp6sp\" (UniqueName: \"kubernetes.io/projected/5edae0d0-8211-41ba-8b4b-154fa2b3f840-kube-api-access-bp6sp\") pod \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.021162    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-data\") pod \"acaf418a-affb-4714-a5ce-5673ed5be68e\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.021196    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/212c33bc-85a6-4ff5-a96a-eda0064fab63-kube-api-access-2wgkx\" (UniqueName: \"kubernetes.io/projected/212c33bc-85a6-4ff5-a96a-eda0064fab63-kube-api-access-2wgkx\") pod \"212c33bc-85a6-4ff5-a96a-eda0064fab63\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.024865    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-workload-socket\") pod \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.024912    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de4f8ca-60d4-443b-9307-e7c763130938-kube-api-access-2fhmh\" (UniqueName: \"kubernetes.io/projected/9de4f8ca-60d4-443b-9307-e7c763130938-kube-api-access-2fhmh\") pod \"9de4f8ca-60d4-443b-9307-e7c763130938\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.024976    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-data\") pod \"212c33bc-85a6-4ff5-a96a-eda0064fab63\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.025009    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-workload-certs\") pod \"acaf418a-affb-4714-a5ce-5673ed5be68e\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.025046    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-podinfo\") pod \"acaf418a-affb-4714-a5ce-5673ed5be68e\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.025097    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-credential-socket\") pod \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.025204    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-envoy\") pod \"acaf418a-affb-4714-a5ce-5673ed5be68e\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.025246    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9de4f8ca-60d4-443b-9307-e7c763130938-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de4f8ca-60d4-443b-9307-e7c763130938-istio-podinfo\") pod \"9de4f8ca-60d4-443b-9307-e7c763130938\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.025272    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-envoy\") pod \"212c33bc-85a6-4ff5-a96a-eda0064fab63\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.025297    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-workload-socket\") pod \"9de4f8ca-60d4-443b-9307-e7c763130938\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.025367    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-data\") pod \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.025427    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/212c33bc-85a6-4ff5-a96a-eda0064fab63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212c33bc-85a6-4ff5-a96a-eda0064fab63-istiod-ca-cert\") pod \"212c33bc-85a6-4ff5-a96a-eda0064fab63\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.025468    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acaf418a-affb-4714-a5ce-5673ed5be68e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acaf418a-affb-4714-a5ce-5673ed5be68e-istiod-ca-cert\") pod \"acaf418a-affb-4714-a5ce-5673ed5be68e\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.025499    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-workload-certs\") pod \"9de4f8ca-60d4-443b-9307-e7c763130938\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.025562    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-envoy\") pod \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.025597    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-workload-socket\") pod \"acaf418a-affb-4714-a5ce-5673ed5be68e\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.025624    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-istio-envoy\") pod \"9de4f8ca-60d4-443b-9307-e7c763130938\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.025656    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-podinfo\") pod \"212c33bc-85a6-4ff5-a96a-eda0064fab63\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.025687    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-token\" (UniqueName: \"kubernetes.io/projected/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-token\") pod \"acaf418a-affb-4714-a5ce-5673ed5be68e\" (UID: \"acaf418a-affb-4714-a5ce-5673ed5be68e\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.025727    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-credential-socket\") pod \"9de4f8ca-60d4-443b-9307-e7c763130938\" (UID: \"9de4f8ca-60d4-443b-9307-e7c763130938\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.025949    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-token\" (UniqueName: \"kubernetes.io/projected/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-token\") pod \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\" (UID: \"5edae0d0-8211-41ba-8b4b-154fa2b3f840\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.025984    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-credential-socket\") pod \"212c33bc-85a6-4ff5-a96a-eda0064fab63\" (UID: \"212c33bc-85a6-4ff5-a96a-eda0064fab63\") "
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.027009    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.027033    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.027048    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.027063    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.027078    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tckst\" (UniqueName: \"kubernetes.io/projected/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd-kube-api-access-tckst\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.029964    5070 scope.go:122] "RemoveContainer" containerID="70c1b4eae07e31e0c089d4f896026fc73ce3527a0c3fd1ea6370da4d5460c15d"
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:16.030352    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70c1b4eae07e31e0c089d4f896026fc73ce3527a0c3fd1ea6370da4d5460c15d\": container with ID starting with 70c1b4eae07e31e0c089d4f896026fc73ce3527a0c3fd1ea6370da4d5460c15d not found: ID does not exist" containerID="70c1b4eae07e31e0c089d4f896026fc73ce3527a0c3fd1ea6370da4d5460c15d"
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.030388    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70c1b4eae07e31e0c089d4f896026fc73ce3527a0c3fd1ea6370da4d5460c15d"} err="failed to get container status \"70c1b4eae07e31e0c089d4f896026fc73ce3527a0c3fd1ea6370da4d5460c15d\": rpc error: code = NotFound desc = could not find container \"70c1b4eae07e31e0c089d4f896026fc73ce3527a0c3fd1ea6370da4d5460c15d\": container with ID starting with 70c1b4eae07e31e0c089d4f896026fc73ce3527a0c3fd1ea6370da4d5460c15d not found: ID does not exist"
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.030409    5070 scope.go:122] "RemoveContainer" containerID="65447f53b80c7570e545e62a80ac6de929e011ddf6aca53a6715a7d0dbbcf33c"
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.021629    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-istio-data" pod "9de4f8ca-60d4-443b-9307-e7c763130938" (UID: "9de4f8ca-60d4-443b-9307-e7c763130938"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.021875    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-workload-certs" pod "212c33bc-85a6-4ff5-a96a-eda0064fab63" (UID: "212c33bc-85a6-4ff5-a96a-eda0064fab63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.022156    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de4f8ca-60d4-443b-9307-e7c763130938-istiod-ca-cert" pod "9de4f8ca-60d4-443b-9307-e7c763130938" (UID: "9de4f8ca-60d4-443b-9307-e7c763130938"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.022403    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-credential-socket" pod "acaf418a-affb-4714-a5ce-5673ed5be68e" (UID: "acaf418a-affb-4714-a5ce-5673ed5be68e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.024505    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istiod-ca-cert" pod "5edae0d0-8211-41ba-8b4b-154fa2b3f840" (UID: "5edae0d0-8211-41ba-8b4b-154fa2b3f840"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.027593    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-data" pod "acaf418a-affb-4714-a5ce-5673ed5be68e" (UID: "acaf418a-affb-4714-a5ce-5673ed5be68e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.030253    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-workload-certs" pod "5edae0d0-8211-41ba-8b4b-154fa2b3f840" (UID: "5edae0d0-8211-41ba-8b4b-154fa2b3f840"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.031683    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-workload-socket" pod "212c33bc-85a6-4ff5-a96a-eda0064fab63" (UID: "212c33bc-85a6-4ff5-a96a-eda0064fab63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.031926    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-data" pod "5edae0d0-8211-41ba-8b4b-154fa2b3f840" (UID: "5edae0d0-8211-41ba-8b4b-154fa2b3f840"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:16.035509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.032187    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-workload-socket" pod "5edae0d0-8211-41ba-8b4b-154fa2b3f840" (UID: "5edae0d0-8211-41ba-8b4b-154fa2b3f840"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:16.037962 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.037936    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de4f8ca-60d4-443b-9307-e7c763130938-istio-token" pod "9de4f8ca-60d4-443b-9307-e7c763130938" (UID: "9de4f8ca-60d4-443b-9307-e7c763130938"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:16.038193 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.038170    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-workload-certs" pod "9de4f8ca-60d4-443b-9307-e7c763130938" (UID: "9de4f8ca-60d4-443b-9307-e7c763130938"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:16.038505 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.038483    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/212c33bc-85a6-4ff5-a96a-eda0064fab63-istiod-ca-cert" pod "212c33bc-85a6-4ff5-a96a-eda0064fab63" (UID: "212c33bc-85a6-4ff5-a96a-eda0064fab63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:16.038870 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.038846    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acaf418a-affb-4714-a5ce-5673ed5be68e-istiod-ca-cert" pod "acaf418a-affb-4714-a5ce-5673ed5be68e" (UID: "acaf418a-affb-4714-a5ce-5673ed5be68e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:16.039582 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.039553    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-istio-envoy" pod "9de4f8ca-60d4-443b-9307-e7c763130938" (UID: "9de4f8ca-60d4-443b-9307-e7c763130938"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:16.039689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.039649    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-token" pod "212c33bc-85a6-4ff5-a96a-eda0064fab63" (UID: "212c33bc-85a6-4ff5-a96a-eda0064fab63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:16.039911 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.039836    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-podinfo" pod "5edae0d0-8211-41ba-8b4b-154fa2b3f840" (UID: "5edae0d0-8211-41ba-8b4b-154fa2b3f840"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:16.041014 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.039939    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-credential-socket" pod "5edae0d0-8211-41ba-8b4b-154fa2b3f840" (UID: "5edae0d0-8211-41ba-8b4b-154fa2b3f840"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:16.041014 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.039964    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de4f8ca-60d4-443b-9307-e7c763130938-kube-api-access-2fhmh" pod "9de4f8ca-60d4-443b-9307-e7c763130938" (UID: "9de4f8ca-60d4-443b-9307-e7c763130938"). InnerVolumeSpecName "kube-api-access-2fhmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:16.041014 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.040320    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-workload-certs" pod "acaf418a-affb-4714-a5ce-5673ed5be68e" (UID: "acaf418a-affb-4714-a5ce-5673ed5be68e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:16.041014 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.040528    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-data" pod "212c33bc-85a6-4ff5-a96a-eda0064fab63" (UID: "212c33bc-85a6-4ff5-a96a-eda0064fab63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:16.041610 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.041204    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-workload-socket" pod "acaf418a-affb-4714-a5ce-5673ed5be68e" (UID: "acaf418a-affb-4714-a5ce-5673ed5be68e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:16.041610 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.041241    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acaf418a-affb-4714-a5ce-5673ed5be68e-kube-api-access-lnkh2" pod "acaf418a-affb-4714-a5ce-5673ed5be68e" (UID: "acaf418a-affb-4714-a5ce-5673ed5be68e"). InnerVolumeSpecName "kube-api-access-lnkh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:16.041610 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.041302    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/212c33bc-85a6-4ff5-a96a-eda0064fab63-kube-api-access-2wgkx" pod "212c33bc-85a6-4ff5-a96a-eda0064fab63" (UID: "212c33bc-85a6-4ff5-a96a-eda0064fab63"). InnerVolumeSpecName "kube-api-access-2wgkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:16.042055 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.041428    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edae0d0-8211-41ba-8b4b-154fa2b3f840-kube-api-access-bp6sp" pod "5edae0d0-8211-41ba-8b4b-154fa2b3f840" (UID: "5edae0d0-8211-41ba-8b4b-154fa2b3f840"). InnerVolumeSpecName "kube-api-access-bp6sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:16.042055 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.041713    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-credential-socket" pod "212c33bc-85a6-4ff5-a96a-eda0064fab63" (UID: "212c33bc-85a6-4ff5-a96a-eda0064fab63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:16.042055 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.041826    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-credential-socket" pod "9de4f8ca-60d4-443b-9307-e7c763130938" (UID: "9de4f8ca-60d4-443b-9307-e7c763130938"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:16.042055 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.041880    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-workload-socket" pod "9de4f8ca-60d4-443b-9307-e7c763130938" (UID: "9de4f8ca-60d4-443b-9307-e7c763130938"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:16.049154 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.042917    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-envoy" pod "5edae0d0-8211-41ba-8b4b-154fa2b3f840" (UID: "5edae0d0-8211-41ba-8b4b-154fa2b3f840"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:16.049154 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.047881    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-token" pod "acaf418a-affb-4714-a5ce-5673ed5be68e" (UID: "acaf418a-affb-4714-a5ce-5673ed5be68e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:16.050839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.050113    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8db8789-e6e5-490d-8a65-5681e647be89" path="/var/lib/kubelet/pods/d8db8789-e6e5-490d-8a65-5681e647be89/volumes"
Nov 03 08:24:16.056379 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.052403    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9de4f8ca-60d4-443b-9307-e7c763130938-istio-podinfo" pod "9de4f8ca-60d4-443b-9307-e7c763130938" (UID: "9de4f8ca-60d4-443b-9307-e7c763130938"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:16.057315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.057160    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-podinfo" pod "acaf418a-affb-4714-a5ce-5673ed5be68e" (UID: "acaf418a-affb-4714-a5ce-5673ed5be68e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:16.057315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.057177    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-envoy" pod "212c33bc-85a6-4ff5-a96a-eda0064fab63" (UID: "212c33bc-85a6-4ff5-a96a-eda0064fab63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:16.057315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.057188    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-podinfo" pod "212c33bc-85a6-4ff5-a96a-eda0064fab63" (UID: "212c33bc-85a6-4ff5-a96a-eda0064fab63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:16.057315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.057282    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-envoy" pod "acaf418a-affb-4714-a5ce-5673ed5be68e" (UID: "acaf418a-affb-4714-a5ce-5673ed5be68e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:16.067887 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.067860    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-token" pod "5edae0d0-8211-41ba-8b4b-154fa2b3f840" (UID: "5edae0d0-8211-41ba-8b4b-154fa2b3f840"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:16.073436 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.073417    5070 scope.go:122] "RemoveContainer" containerID="65447f53b80c7570e545e62a80ac6de929e011ddf6aca53a6715a7d0dbbcf33c"
Nov 03 08:24:16.073825 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:16.073803    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65447f53b80c7570e545e62a80ac6de929e011ddf6aca53a6715a7d0dbbcf33c\": container with ID starting with 65447f53b80c7570e545e62a80ac6de929e011ddf6aca53a6715a7d0dbbcf33c not found: ID does not exist" containerID="65447f53b80c7570e545e62a80ac6de929e011ddf6aca53a6715a7d0dbbcf33c"
Nov 03 08:24:16.073907 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.073831    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65447f53b80c7570e545e62a80ac6de929e011ddf6aca53a6715a7d0dbbcf33c"} err="failed to get container status \"65447f53b80c7570e545e62a80ac6de929e011ddf6aca53a6715a7d0dbbcf33c\": rpc error: code = NotFound desc = could not find container \"65447f53b80c7570e545e62a80ac6de929e011ddf6aca53a6715a7d0dbbcf33c\": container with ID starting with 65447f53b80c7570e545e62a80ac6de929e011ddf6aca53a6715a7d0dbbcf33c not found: ID does not exist"
Nov 03 08:24:16.073907 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.073847    5070 scope.go:122] "RemoveContainer" containerID="906ca3f54a6077a62a9475c973ab73f82b8058b800fa080208b163d7c16361e5"
Nov 03 08:24:16.086550 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.086525    5070 scope.go:122] "RemoveContainer" containerID="906ca3f54a6077a62a9475c973ab73f82b8058b800fa080208b163d7c16361e5"
Nov 03 08:24:16.087151 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:16.087119    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"906ca3f54a6077a62a9475c973ab73f82b8058b800fa080208b163d7c16361e5\": container with ID starting with 906ca3f54a6077a62a9475c973ab73f82b8058b800fa080208b163d7c16361e5 not found: ID does not exist" containerID="906ca3f54a6077a62a9475c973ab73f82b8058b800fa080208b163d7c16361e5"
Nov 03 08:24:16.087358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.087160    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906ca3f54a6077a62a9475c973ab73f82b8058b800fa080208b163d7c16361e5"} err="failed to get container status \"906ca3f54a6077a62a9475c973ab73f82b8058b800fa080208b163d7c16361e5\": rpc error: code = NotFound desc = could not find container \"906ca3f54a6077a62a9475c973ab73f82b8058b800fa080208b163d7c16361e5\": container with ID starting with 906ca3f54a6077a62a9475c973ab73f82b8058b800fa080208b163d7c16361e5 not found: ID does not exist"
Nov 03 08:24:16.087358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.087181    5070 scope.go:122] "RemoveContainer" containerID="11a078b3ccf228e563e1e3b67d99f29a2327698ee3cc23f7dc65a574eccf3f83"
Nov 03 08:24:16.088246 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.088222    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"]
Nov 03 08:24:16.092888 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.092862    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-zk92r"]
Nov 03 08:24:16.099502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.099481    5070 scope.go:122] "RemoveContainer" containerID="9bd4a10841b8474e8baebcb447545c93c6cb1b79d710ac3e880d8227edf1f286"
Nov 03 08:24:16.110328 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.110297    5070 scope.go:122] "RemoveContainer" containerID="11a078b3ccf228e563e1e3b67d99f29a2327698ee3cc23f7dc65a574eccf3f83"
Nov 03 08:24:16.110563 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:16.110543    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a078b3ccf228e563e1e3b67d99f29a2327698ee3cc23f7dc65a574eccf3f83\": container with ID starting with 11a078b3ccf228e563e1e3b67d99f29a2327698ee3cc23f7dc65a574eccf3f83 not found: ID does not exist" containerID="11a078b3ccf228e563e1e3b67d99f29a2327698ee3cc23f7dc65a574eccf3f83"
Nov 03 08:24:16.110606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.110571    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a078b3ccf228e563e1e3b67d99f29a2327698ee3cc23f7dc65a574eccf3f83"} err="failed to get container status \"11a078b3ccf228e563e1e3b67d99f29a2327698ee3cc23f7dc65a574eccf3f83\": rpc error: code = NotFound desc = could not find container \"11a078b3ccf228e563e1e3b67d99f29a2327698ee3cc23f7dc65a574eccf3f83\": container with ID starting with 11a078b3ccf228e563e1e3b67d99f29a2327698ee3cc23f7dc65a574eccf3f83 not found: ID does not exist"
Nov 03 08:24:16.110606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.110587    5070 scope.go:122] "RemoveContainer" containerID="9bd4a10841b8474e8baebcb447545c93c6cb1b79d710ac3e880d8227edf1f286"
Nov 03 08:24:16.110849 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:16.110830    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bd4a10841b8474e8baebcb447545c93c6cb1b79d710ac3e880d8227edf1f286\": container with ID starting with 9bd4a10841b8474e8baebcb447545c93c6cb1b79d710ac3e880d8227edf1f286 not found: ID does not exist" containerID="9bd4a10841b8474e8baebcb447545c93c6cb1b79d710ac3e880d8227edf1f286"
Nov 03 08:24:16.110984 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.110850    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bd4a10841b8474e8baebcb447545c93c6cb1b79d710ac3e880d8227edf1f286"} err="failed to get container status \"9bd4a10841b8474e8baebcb447545c93c6cb1b79d710ac3e880d8227edf1f286\": rpc error: code = NotFound desc = could not find container \"9bd4a10841b8474e8baebcb447545c93c6cb1b79d710ac3e880d8227edf1f286\": container with ID starting with 9bd4a10841b8474e8baebcb447545c93c6cb1b79d710ac3e880d8227edf1f286 not found: ID does not exist"
Nov 03 08:24:16.110984 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.110864    5070 scope.go:122] "RemoveContainer" containerID="a967eb664dd1a02bc3f7caa6e51c62911ebadfe4e74a25d5e7a237a847450b90"
Nov 03 08:24:16.121981 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.121964    5070 scope.go:122] "RemoveContainer" containerID="a967eb664dd1a02bc3f7caa6e51c62911ebadfe4e74a25d5e7a237a847450b90"
Nov 03 08:24:16.122230 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:16.122211    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a967eb664dd1a02bc3f7caa6e51c62911ebadfe4e74a25d5e7a237a847450b90\": container with ID starting with a967eb664dd1a02bc3f7caa6e51c62911ebadfe4e74a25d5e7a237a847450b90 not found: ID does not exist" containerID="a967eb664dd1a02bc3f7caa6e51c62911ebadfe4e74a25d5e7a237a847450b90"
Nov 03 08:24:16.122287 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.122240    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a967eb664dd1a02bc3f7caa6e51c62911ebadfe4e74a25d5e7a237a847450b90"} err="failed to get container status \"a967eb664dd1a02bc3f7caa6e51c62911ebadfe4e74a25d5e7a237a847450b90\": rpc error: code = NotFound desc = could not find container \"a967eb664dd1a02bc3f7caa6e51c62911ebadfe4e74a25d5e7a237a847450b90\": container with ID starting with a967eb664dd1a02bc3f7caa6e51c62911ebadfe4e74a25d5e7a237a847450b90 not found: ID does not exist"
Nov 03 08:24:16.128510 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128483    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.128510 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128509    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.128649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128523    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.128649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128536    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.128649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128545    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.128649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128554    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de4f8ca-60d4-443b-9307-e7c763130938-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.128649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128563    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.128649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128571    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de4f8ca-60d4-443b-9307-e7c763130938-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.128649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128583    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.128649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128597    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.128649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128606    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnkh2\" (UniqueName: \"kubernetes.io/projected/acaf418a-affb-4714-a5ce-5673ed5be68e-kube-api-access-lnkh2\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.128649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128615    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.128649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128623    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.128649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128632    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp6sp\" (UniqueName: \"kubernetes.io/projected/5edae0d0-8211-41ba-8b4b-154fa2b3f840-kube-api-access-bp6sp\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.128649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128641    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.128649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128650    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wgkx\" (UniqueName: \"kubernetes.io/projected/212c33bc-85a6-4ff5-a96a-eda0064fab63-kube-api-access-2wgkx\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.129257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128661    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.129257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128675    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fhmh\" (UniqueName: \"kubernetes.io/projected/9de4f8ca-60d4-443b-9307-e7c763130938-kube-api-access-2fhmh\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.129257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128683    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.129257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128691    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.129257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128700    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.129257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128708    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.129257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128717    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.129257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128725    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de4f8ca-60d4-443b-9307-e7c763130938-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.129257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128739    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.129257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128752    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.129257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128776    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.129257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128784    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/212c33bc-85a6-4ff5-a96a-eda0064fab63-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.129257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128794    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acaf418a-affb-4714-a5ce-5673ed5be68e-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.129257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128802    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.129257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128810    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5edae0d0-8211-41ba-8b4b-154fa2b3f840-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.129257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128818    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acaf418a-affb-4714-a5ce-5673ed5be68e-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.129257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128830    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.129257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128842    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/212c33bc-85a6-4ff5-a96a-eda0064fab63-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.129257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128850    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acaf418a-affb-4714-a5ce-5673ed5be68e-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.129257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.128858    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de4f8ca-60d4-443b-9307-e7c763130938-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:16.327815 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.327790    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"]
Nov 03 08:24:16.332317 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.332293    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-btrmk"]
Nov 03 08:24:16.337096 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.337067    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"]
Nov 03 08:24:16.340582 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.340564    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-8pqss"]
Nov 03 08:24:16.349009 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.348990    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"]
Nov 03 08:24:16.355478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.355460    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-7xk2j"]
Nov 03 08:24:16.363845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.363826    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"]
Nov 03 08:24:16.369139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:16.369118    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9b559e-openshift-default-7dd667487c-hjpp8"]
Nov 03 08:24:18.031040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:18.031004    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="212c33bc-85a6-4ff5-a96a-eda0064fab63" path="/var/lib/kubelet/pods/212c33bc-85a6-4ff5-a96a-eda0064fab63/volumes"
Nov 03 08:24:18.032532 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:18.032504    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5edae0d0-8211-41ba-8b4b-154fa2b3f840" path="/var/lib/kubelet/pods/5edae0d0-8211-41ba-8b4b-154fa2b3f840/volumes"
Nov 03 08:24:18.033472 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:18.033452    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9de4f8ca-60d4-443b-9307-e7c763130938" path="/var/lib/kubelet/pods/9de4f8ca-60d4-443b-9307-e7c763130938/volumes"
Nov 03 08:24:18.034775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:18.034743    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" path="/var/lib/kubelet/pods/acaf418a-affb-4714-a5ce-5673ed5be68e/volumes"
Nov 03 08:24:18.035667 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:18.035647    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" path="/var/lib/kubelet/pods/bb569224-4a9c-4da1-b5c4-cb6ea74b68bd/volumes"
Nov 03 08:24:19.042891 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.042864    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:24:19.059811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.059778    5070 generic.go:356] "Generic (PLEG): container finished" podID="d7faa6ae-27c1-4a98-8c83-4a3fb1669835" containerID="020923369facd8800e378661dd8b7b52e1428e0e7aa851b1044a81716cc139f8" exitCode=0
Nov 03 08:24:19.059977 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.059847    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"
Nov 03 08:24:19.059977 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.059872    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx" event={"ID":"d7faa6ae-27c1-4a98-8c83-4a3fb1669835","Type":"ContainerDied","Data":"020923369facd8800e378661dd8b7b52e1428e0e7aa851b1044a81716cc139f8"}
Nov 03 08:24:19.059977 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.059943    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx" event={"ID":"d7faa6ae-27c1-4a98-8c83-4a3fb1669835","Type":"ContainerDied","Data":"9d0091b34ea1d8cc7285fee5f2a01d7039b24bc0fb8c8d6217df2b863bc63f1d"}
Nov 03 08:24:19.060174 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.059978    5070 scope.go:122] "RemoveContainer" containerID="020923369facd8800e378661dd8b7b52e1428e0e7aa851b1044a81716cc139f8"
Nov 03 08:24:19.066301 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.065895    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-kube-api-access-hdxn7\" (UniqueName: \"kubernetes.io/projected/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-kube-api-access-hdxn7\") pod \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") "
Nov 03 08:24:19.066301 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.065987    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-workload-certs\") pod \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") "
Nov 03 08:24:19.066301 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.066023    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-envoy\") pod \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") "
Nov 03 08:24:19.066301 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.066108    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-data\") pod \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") "
Nov 03 08:24:19.066301 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.066170    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-credential-socket\") pod \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") "
Nov 03 08:24:19.066664 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.066332    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-token\" (UniqueName: \"kubernetes.io/projected/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-token\") pod \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") "
Nov 03 08:24:19.066664 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.066386    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-workload-socket\") pod \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") "
Nov 03 08:24:19.066664 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.066420    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-podinfo\") pod \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") "
Nov 03 08:24:19.066664 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.066449    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istiod-ca-cert\") pod \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\" (UID: \"d7faa6ae-27c1-4a98-8c83-4a3fb1669835\") "
Nov 03 08:24:19.067894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.067856    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-credential-socket" pod "d7faa6ae-27c1-4a98-8c83-4a3fb1669835" (UID: "d7faa6ae-27c1-4a98-8c83-4a3fb1669835"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:19.068322 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.068304    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istiod-ca-cert" pod "d7faa6ae-27c1-4a98-8c83-4a3fb1669835" (UID: "d7faa6ae-27c1-4a98-8c83-4a3fb1669835"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:19.068901 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.068868    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-workload-socket" pod "d7faa6ae-27c1-4a98-8c83-4a3fb1669835" (UID: "d7faa6ae-27c1-4a98-8c83-4a3fb1669835"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:19.081797 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.071550    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-envoy" pod "d7faa6ae-27c1-4a98-8c83-4a3fb1669835" (UID: "d7faa6ae-27c1-4a98-8c83-4a3fb1669835"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:19.081797 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.071668    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-kube-api-access-hdxn7" pod "d7faa6ae-27c1-4a98-8c83-4a3fb1669835" (UID: "d7faa6ae-27c1-4a98-8c83-4a3fb1669835"). InnerVolumeSpecName "kube-api-access-hdxn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:19.081797 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.080902    5070 scope.go:122] "RemoveContainer" containerID="020923369facd8800e378661dd8b7b52e1428e0e7aa851b1044a81716cc139f8"
Nov 03 08:24:19.081797 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:19.081233    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"020923369facd8800e378661dd8b7b52e1428e0e7aa851b1044a81716cc139f8\": container with ID starting with 020923369facd8800e378661dd8b7b52e1428e0e7aa851b1044a81716cc139f8 not found: ID does not exist" containerID="020923369facd8800e378661dd8b7b52e1428e0e7aa851b1044a81716cc139f8"
Nov 03 08:24:19.081797 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.081268    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020923369facd8800e378661dd8b7b52e1428e0e7aa851b1044a81716cc139f8"} err="failed to get container status \"020923369facd8800e378661dd8b7b52e1428e0e7aa851b1044a81716cc139f8\": rpc error: code = NotFound desc = could not find container \"020923369facd8800e378661dd8b7b52e1428e0e7aa851b1044a81716cc139f8\": container with ID starting with 020923369facd8800e378661dd8b7b52e1428e0e7aa851b1044a81716cc139f8 not found: ID does not exist"
Nov 03 08:24:19.096294 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.096226    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-data" pod "d7faa6ae-27c1-4a98-8c83-4a3fb1669835" (UID: "d7faa6ae-27c1-4a98-8c83-4a3fb1669835"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:19.096682 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.096654    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-workload-certs" pod "d7faa6ae-27c1-4a98-8c83-4a3fb1669835" (UID: "d7faa6ae-27c1-4a98-8c83-4a3fb1669835"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:19.098038 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.098010    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-podinfo" pod "d7faa6ae-27c1-4a98-8c83-4a3fb1669835" (UID: "d7faa6ae-27c1-4a98-8c83-4a3fb1669835"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:19.098616 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.098595    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-token" pod "d7faa6ae-27c1-4a98-8c83-4a3fb1669835" (UID: "d7faa6ae-27c1-4a98-8c83-4a3fb1669835"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:19.168025 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.167998    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:19.168025 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.168024    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:19.168025 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.168033    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:19.168305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.168042    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:19.168305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.168051    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:19.168305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.168059    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdxn7\" (UniqueName: \"kubernetes.io/projected/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-kube-api-access-hdxn7\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:19.168305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.168069    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:19.168305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.168077    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:19.168305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.168087    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7faa6ae-27c1-4a98-8c83-4a3fb1669835-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:19.383792 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.383750    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"]
Nov 03 08:24:19.395260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:19.395236    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-76ef13-openshift-default-5f6c559897-d6pgx"]
Nov 03 08:24:20.027876 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:20.027789    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7faa6ae-27c1-4a98-8c83-4a3fb1669835" path="/var/lib/kubelet/pods/d7faa6ae-27c1-4a98-8c83-4a3fb1669835/volumes"
Nov 03 08:24:22.229188 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:22.229152    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:24:22.484796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:22.479926    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"]
Nov 03 08:24:22.484796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:22.480209    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929" podUID="f813310f-254a-4b89-940c-06d7b335a112" containerName="istio-proxy" containerID="cri-o://d67b2683a3325f8a6b25d5aacdac5136a40650214806444b57b9c707da90d03f" gracePeriod=30
Nov 03 08:24:22.495024 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:22.493567    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"]
Nov 03 08:24:22.495024 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:22.493915    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm" podUID="0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" containerName="istio-proxy" containerID="cri-o://c0dac5751a2dd2612c2e5ab4921f8f6c0fe90caa76eeee71bb742ad563d7e24a" gracePeriod=30
Nov 03 08:24:22.532893 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:22.532864    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"]
Nov 03 08:24:22.533331 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:22.533302    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd" podUID="c3d60ea0-edd4-4c91-87be-9b414765dd7c" containerName="istio-proxy" containerID="cri-o://95c97a02c5b2b0d10a4ad46e0a3d7ad73a3e1a56ccb837840cf4f6983beea19e" gracePeriod=30
Nov 03 08:24:22.539880 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:22.539858    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"]
Nov 03 08:24:22.540085 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:22.540059    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm" podUID="5e519992-15af-4d72-ba82-190c90ab0aa8" containerName="istio-proxy" containerID="cri-o://9c93261828483c8ee6e805de46f8a2dc36f1a94de5507e5444255362cf4b45f6" gracePeriod=30
Nov 03 08:24:23.109059 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.109029    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2"]
Nov 03 08:24:23.109693 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.109679    5070 cpu_manager.go:414] "containerMap: removing container" podUID="d7faa6ae-27c1-4a98-8c83-4a3fb1669835" containerName="istio-proxy"
Nov 03 08:24:23.109745 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.109696    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7faa6ae-27c1-4a98-8c83-4a3fb1669835" containerName="istio-proxy"
Nov 03 08:24:23.109745 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.109711    5070 cpu_manager.go:414] "containerMap: removing container" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy"
Nov 03 08:24:23.109745 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.109717    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy"
Nov 03 08:24:23.109745 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.109741    5070 cpu_manager.go:414] "containerMap: removing container" podUID="9de4f8ca-60d4-443b-9307-e7c763130938" containerName="istio-proxy"
Nov 03 08:24:23.109745 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.109745    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9de4f8ca-60d4-443b-9307-e7c763130938" containerName="istio-proxy"
Nov 03 08:24:23.109926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.109754    5070 cpu_manager.go:414] "containerMap: removing container" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy"
Nov 03 08:24:23.109926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.109774    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy"
Nov 03 08:24:23.109926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.109793    5070 cpu_manager.go:414] "containerMap: removing container" podUID="bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" containerName="istio-proxy"
Nov 03 08:24:23.109926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.109798    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" containerName="istio-proxy"
Nov 03 08:24:23.109926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.109823    5070 cpu_manager.go:414] "containerMap: removing container" podUID="212c33bc-85a6-4ff5-a96a-eda0064fab63" containerName="istio-proxy"
Nov 03 08:24:23.109926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.109828    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="212c33bc-85a6-4ff5-a96a-eda0064fab63" containerName="istio-proxy"
Nov 03 08:24:23.109926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.109835    5070 cpu_manager.go:414] "containerMap: removing container" podUID="803348c9-5802-4b5a-836d-462fb6ebc404" containerName="revoke-keys"
Nov 03 08:24:23.109926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.109839    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="803348c9-5802-4b5a-836d-462fb6ebc404" containerName="revoke-keys"
Nov 03 08:24:23.109926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.109857    5070 cpu_manager.go:414] "containerMap: removing container" podUID="5edae0d0-8211-41ba-8b4b-154fa2b3f840" containerName="istio-proxy"
Nov 03 08:24:23.109926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.109864    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5edae0d0-8211-41ba-8b4b-154fa2b3f840" containerName="istio-proxy"
Nov 03 08:24:23.110295 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.109968    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7faa6ae-27c1-4a98-8c83-4a3fb1669835" containerName="istio-proxy"
Nov 03 08:24:23.110295 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.109979    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy"
Nov 03 08:24:23.110295 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.109991    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="9de4f8ca-60d4-443b-9307-e7c763130938" containerName="istio-proxy"
Nov 03 08:24:23.110295 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.109996    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="acaf418a-affb-4714-a5ce-5673ed5be68e" containerName="istio-proxy"
Nov 03 08:24:23.110295 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.110002    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb569224-4a9c-4da1-b5c4-cb6ea74b68bd" containerName="istio-proxy"
Nov 03 08:24:23.110295 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.110017    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="212c33bc-85a6-4ff5-a96a-eda0064fab63" containerName="istio-proxy"
Nov 03 08:24:23.110295 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.110024    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="803348c9-5802-4b5a-836d-462fb6ebc404" containerName="revoke-keys"
Nov 03 08:24:23.110295 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.110031    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="5edae0d0-8211-41ba-8b4b-154fa2b3f840" containerName="istio-proxy"
Nov 03 08:24:23.115132 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.115105    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2"
Nov 03 08:24:23.118545 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.118522    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:24:23.118879 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.118853    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:24:23.119011 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.118995    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:24:23.119089 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.119043    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:24:23.121965 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.121942    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2"]
Nov 03 08:24:23.185652 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.185618    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2484ae5d-667e-4ee2-a1fa-01c61ded3d45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2\" (UID: \"2484ae5d-667e-4ee2-a1fa-01c61ded3d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2"
Nov 03 08:24:23.185877 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.185856    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2484ae5d-667e-4ee2-a1fa-01c61ded3d45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2\" (UID: \"2484ae5d-667e-4ee2-a1fa-01c61ded3d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2"
Nov 03 08:24:23.186016 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.185998    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkzfj\" (UniqueName: \"kubernetes.io/projected/2484ae5d-667e-4ee2-a1fa-01c61ded3d45-kube-api-access-dkzfj\") pod \"maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2\" (UID: \"2484ae5d-667e-4ee2-a1fa-01c61ded3d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2"
Nov 03 08:24:23.287422 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.287392    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2484ae5d-667e-4ee2-a1fa-01c61ded3d45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2\" (UID: \"2484ae5d-667e-4ee2-a1fa-01c61ded3d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2"
Nov 03 08:24:23.287813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.287488    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkzfj\" (UniqueName: \"kubernetes.io/projected/2484ae5d-667e-4ee2-a1fa-01c61ded3d45-kube-api-access-dkzfj\") pod \"maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2\" (UID: \"2484ae5d-667e-4ee2-a1fa-01c61ded3d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2"
Nov 03 08:24:23.287813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.287607    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2484ae5d-667e-4ee2-a1fa-01c61ded3d45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2\" (UID: \"2484ae5d-667e-4ee2-a1fa-01c61ded3d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2"
Nov 03 08:24:23.288292 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.288274    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2484ae5d-667e-4ee2-a1fa-01c61ded3d45-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2\" (UID: \"2484ae5d-667e-4ee2-a1fa-01c61ded3d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2"
Nov 03 08:24:23.289627 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.289608    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2484ae5d-667e-4ee2-a1fa-01c61ded3d45-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2\" (UID: \"2484ae5d-667e-4ee2-a1fa-01c61ded3d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2"
Nov 03 08:24:23.295152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.295129    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkzfj\" (UniqueName: \"kubernetes.io/projected/2484ae5d-667e-4ee2-a1fa-01c61ded3d45-kube-api-access-dkzfj\") pod \"maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2\" (UID: \"2484ae5d-667e-4ee2-a1fa-01c61ded3d45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2"
Nov 03 08:24:23.431962 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.431907    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2"
Nov 03 08:24:23.713630 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:23.713604    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2"]
Nov 03 08:24:23.713937 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:24:23.713915    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2484ae5d_667e_4ee2_a1fa_01c61ded3d45.slice/crio-d165f91556e6442741762617ffea0c7bb7770523f30ba2457e3ce79a5df8643d WatchSource:0}: Error finding container d165f91556e6442741762617ffea0c7bb7770523f30ba2457e3ce79a5df8643d: Status 404 returned error can't find the container with id d165f91556e6442741762617ffea0c7bb7770523f30ba2457e3ce79a5df8643d
Nov 03 08:24:24.178997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:24.178960    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2" event={"ID":"2484ae5d-667e-4ee2-a1fa-01c61ded3d45","Type":"ContainerStarted","Data":"2da2634436e9ba0c3aa4851cc650c7c380d558837d1e07443abb0127d008625f"}
Nov 03 08:24:24.179169 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:24.179017    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2" event={"ID":"2484ae5d-667e-4ee2-a1fa-01c61ded3d45","Type":"ContainerStarted","Data":"d165f91556e6442741762617ffea0c7bb7770523f30ba2457e3ce79a5df8643d"}
Nov 03 08:24:24.192291 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:24.192242    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2" podStartSLOduration=1.192228905 podStartE2EDuration="1.192228905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:24:24.191983686 +0000 UTC m=+1862.306447617" watchObservedRunningTime="2026-11-03 08:24:24.192228905 +0000 UTC m=+1862.306692837"
Nov 03 08:24:25.188651 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:25.188616    5070 generic.go:356] "Generic (PLEG): container finished" podID="2484ae5d-667e-4ee2-a1fa-01c61ded3d45" containerID="2da2634436e9ba0c3aa4851cc650c7c380d558837d1e07443abb0127d008625f" exitCode=0
Nov 03 08:24:25.189045 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:25.188704    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2" event={"ID":"2484ae5d-667e-4ee2-a1fa-01c61ded3d45","Type":"ContainerDied","Data":"2da2634436e9ba0c3aa4851cc650c7c380d558837d1e07443abb0127d008625f"}
Nov 03 08:24:26.399676 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:26.399651    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2"
Nov 03 08:24:26.537821 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:26.537729    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2484ae5d-667e-4ee2-a1fa-01c61ded3d45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2484ae5d-667e-4ee2-a1fa-01c61ded3d45-maas-api-openshift-ca\") pod \"2484ae5d-667e-4ee2-a1fa-01c61ded3d45\" (UID: \"2484ae5d-667e-4ee2-a1fa-01c61ded3d45\") "
Nov 03 08:24:26.537963 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:26.537828    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2484ae5d-667e-4ee2-a1fa-01c61ded3d45-kube-api-access-dkzfj\" (UniqueName: \"kubernetes.io/projected/2484ae5d-667e-4ee2-a1fa-01c61ded3d45-kube-api-access-dkzfj\") pod \"2484ae5d-667e-4ee2-a1fa-01c61ded3d45\" (UID: \"2484ae5d-667e-4ee2-a1fa-01c61ded3d45\") "
Nov 03 08:24:26.537963 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:26.537906    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2484ae5d-667e-4ee2-a1fa-01c61ded3d45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2484ae5d-667e-4ee2-a1fa-01c61ded3d45-maas-api-xks-ca\") pod \"2484ae5d-667e-4ee2-a1fa-01c61ded3d45\" (UID: \"2484ae5d-667e-4ee2-a1fa-01c61ded3d45\") "
Nov 03 08:24:26.538121 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:26.538094    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2484ae5d-667e-4ee2-a1fa-01c61ded3d45-maas-api-openshift-ca" pod "2484ae5d-667e-4ee2-a1fa-01c61ded3d45" (UID: "2484ae5d-667e-4ee2-a1fa-01c61ded3d45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:26.538487 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:26.538464    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2484ae5d-667e-4ee2-a1fa-01c61ded3d45-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:26.539980 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:26.539960    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2484ae5d-667e-4ee2-a1fa-01c61ded3d45-maas-api-xks-ca" pod "2484ae5d-667e-4ee2-a1fa-01c61ded3d45" (UID: "2484ae5d-667e-4ee2-a1fa-01c61ded3d45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:24:26.540081 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:26.540055    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2484ae5d-667e-4ee2-a1fa-01c61ded3d45-kube-api-access-dkzfj" pod "2484ae5d-667e-4ee2-a1fa-01c61ded3d45" (UID: "2484ae5d-667e-4ee2-a1fa-01c61ded3d45"). InnerVolumeSpecName "kube-api-access-dkzfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:26.640580 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:26.640547    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2484ae5d-667e-4ee2-a1fa-01c61ded3d45-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:26.640580 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:26.640580    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkzfj\" (UniqueName: \"kubernetes.io/projected/2484ae5d-667e-4ee2-a1fa-01c61ded3d45-kube-api-access-dkzfj\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:27.207827 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.207798    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2"
Nov 03 08:24:27.207827 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.207808    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2" event={"ID":"2484ae5d-667e-4ee2-a1fa-01c61ded3d45","Type":"ContainerDied","Data":"d165f91556e6442741762617ffea0c7bb7770523f30ba2457e3ce79a5df8643d"}
Nov 03 08:24:27.208033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.207856    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d165f91556e6442741762617ffea0c7bb7770523f30ba2457e3ce79a5df8643d"
Nov 03 08:24:27.854575 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.854552    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:24:27.923867 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.923836    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:24:27.930586 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.930560    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:24:27.958825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.958392    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istiod-ca-cert\") pod \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") "
Nov 03 08:24:27.961235 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.961206    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istiod-ca-cert" pod "c3d60ea0-edd4-4c91-87be-9b414765dd7c" (UID: "c3d60ea0-edd4-4c91-87be-9b414765dd7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:27.961434 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.961310    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-podinfo\") pod \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") "
Nov 03 08:24:27.961752 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.961735    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d60ea0-edd4-4c91-87be-9b414765dd7c-kube-api-access-zbhvx\" (UniqueName: \"kubernetes.io/projected/c3d60ea0-edd4-4c91-87be-9b414765dd7c-kube-api-access-zbhvx\") pod \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") "
Nov 03 08:24:27.962413 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.962386    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-envoy\") pod \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") "
Nov 03 08:24:27.962516 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.962432    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-workload-socket\") pod \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") "
Nov 03 08:24:27.962589 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.962543    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-token\" (UniqueName: \"kubernetes.io/projected/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-token\") pod \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") "
Nov 03 08:24:27.962643 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.962608    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-workload-certs\") pod \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") "
Nov 03 08:24:27.962703 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.962652    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-credential-socket\") pod \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") "
Nov 03 08:24:27.962703 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.962685    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-data\") pod \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\" (UID: \"c3d60ea0-edd4-4c91-87be-9b414765dd7c\") "
Nov 03 08:24:27.963635 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.963457    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-workload-certs" pod "c3d60ea0-edd4-4c91-87be-9b414765dd7c" (UID: "c3d60ea0-edd4-4c91-87be-9b414765dd7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:27.963635 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.963587    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:27.963635 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.963608    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:27.963988 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.963699    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-data" pod "c3d60ea0-edd4-4c91-87be-9b414765dd7c" (UID: "c3d60ea0-edd4-4c91-87be-9b414765dd7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:27.964070 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.963979    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-workload-socket" pod "c3d60ea0-edd4-4c91-87be-9b414765dd7c" (UID: "c3d60ea0-edd4-4c91-87be-9b414765dd7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:27.964138 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.964106    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-credential-socket" pod "c3d60ea0-edd4-4c91-87be-9b414765dd7c" (UID: "c3d60ea0-edd4-4c91-87be-9b414765dd7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:27.968040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.967878    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d60ea0-edd4-4c91-87be-9b414765dd7c-kube-api-access-zbhvx" pod "c3d60ea0-edd4-4c91-87be-9b414765dd7c" (UID: "c3d60ea0-edd4-4c91-87be-9b414765dd7c"). InnerVolumeSpecName "kube-api-access-zbhvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:27.968040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.967948    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-token" pod "c3d60ea0-edd4-4c91-87be-9b414765dd7c" (UID: "c3d60ea0-edd4-4c91-87be-9b414765dd7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:27.968299 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.968269    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-podinfo" pod "c3d60ea0-edd4-4c91-87be-9b414765dd7c" (UID: "c3d60ea0-edd4-4c91-87be-9b414765dd7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:27.968570 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.968551    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-envoy" pod "c3d60ea0-edd4-4c91-87be-9b414765dd7c" (UID: "c3d60ea0-edd4-4c91-87be-9b414765dd7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:27.977338 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:27.976968    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:24:28.064708 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.064644    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f813310f-254a-4b89-940c-06d7b335a112-kube-api-access-jjxdm\" (UniqueName: \"kubernetes.io/projected/f813310f-254a-4b89-940c-06d7b335a112-kube-api-access-jjxdm\") pod \"f813310f-254a-4b89-940c-06d7b335a112\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") "
Nov 03 08:24:28.064708 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.064686    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-workload-socket\") pod \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") "
Nov 03 08:24:28.064960 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.064725    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istiod-ca-cert\") pod \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") "
Nov 03 08:24:28.064960 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.064754    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-token\" (UniqueName: \"kubernetes.io/projected/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-token\") pod \"5e519992-15af-4d72-ba82-190c90ab0aa8\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") "
Nov 03 08:24:28.064960 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.064859    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-credential-socket\") pod \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") "
Nov 03 08:24:28.064960 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.064897    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-token\" (UniqueName: \"kubernetes.io/projected/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-token\") pod \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") "
Nov 03 08:24:28.064960 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.064932    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f813310f-254a-4b89-940c-06d7b335a112-istio-token\" (UniqueName: \"kubernetes.io/projected/f813310f-254a-4b89-940c-06d7b335a112-istio-token\") pod \"f813310f-254a-4b89-940c-06d7b335a112\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") "
Nov 03 08:24:28.065260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.064967    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f813310f-254a-4b89-940c-06d7b335a112-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f813310f-254a-4b89-940c-06d7b335a112-istio-podinfo\") pod \"f813310f-254a-4b89-940c-06d7b335a112\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") "
Nov 03 08:24:28.065260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065002    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-credential-socket\") pod \"5e519992-15af-4d72-ba82-190c90ab0aa8\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") "
Nov 03 08:24:28.065260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065075    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-workload-socket" pod "0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" (UID: "0b4af5de-70dc-4fb5-b542-b4ccbd7c709d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:28.065260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065095    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-istio-data\") pod \"f813310f-254a-4b89-940c-06d7b335a112\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") "
Nov 03 08:24:28.065260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065135    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istiod-ca-cert" pod "0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" (UID: "0b4af5de-70dc-4fb5-b542-b4ccbd7c709d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:28.065260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065176    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-workload-socket\") pod \"f813310f-254a-4b89-940c-06d7b335a112\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") "
Nov 03 08:24:28.065260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065234    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-credential-socket\") pod \"f813310f-254a-4b89-940c-06d7b335a112\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") "
Nov 03 08:24:28.065260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065262    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-data\") pod \"5e519992-15af-4d72-ba82-190c90ab0aa8\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") "
Nov 03 08:24:28.065776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065344    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-podinfo\") pod \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") "
Nov 03 08:24:28.065776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065391    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-workload-socket\") pod \"5e519992-15af-4d72-ba82-190c90ab0aa8\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") "
Nov 03 08:24:28.065776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065433    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-envoy\") pod \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") "
Nov 03 08:24:28.065776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065464    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e519992-15af-4d72-ba82-190c90ab0aa8-kube-api-access-gpx2h\" (UniqueName: \"kubernetes.io/projected/5e519992-15af-4d72-ba82-190c90ab0aa8-kube-api-access-gpx2h\") pod \"5e519992-15af-4d72-ba82-190c90ab0aa8\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") "
Nov 03 08:24:28.065776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065459    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-credential-socket" pod "5e519992-15af-4d72-ba82-190c90ab0aa8" (UID: "5e519992-15af-4d72-ba82-190c90ab0aa8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:28.065776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065507    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-data\") pod \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") "
Nov 03 08:24:28.065776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065564    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-workload-socket" pod "f813310f-254a-4b89-940c-06d7b335a112" (UID: "f813310f-254a-4b89-940c-06d7b335a112"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:28.065776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065583    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-workload-certs\") pod \"5e519992-15af-4d72-ba82-190c90ab0aa8\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") "
Nov 03 08:24:28.065776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065650    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e519992-15af-4d72-ba82-190c90ab0aa8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e519992-15af-4d72-ba82-190c90ab0aa8-istiod-ca-cert\") pod \"5e519992-15af-4d72-ba82-190c90ab0aa8\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") "
Nov 03 08:24:28.065776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065660    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-istio-data" pod "f813310f-254a-4b89-940c-06d7b335a112" (UID: "f813310f-254a-4b89-940c-06d7b335a112"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:28.065776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065697    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f813310f-254a-4b89-940c-06d7b335a112-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f813310f-254a-4b89-940c-06d7b335a112-istiod-ca-cert\") pod \"f813310f-254a-4b89-940c-06d7b335a112\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") "
Nov 03 08:24:28.065776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065772    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-envoy\") pod \"5e519992-15af-4d72-ba82-190c90ab0aa8\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") "
Nov 03 08:24:28.066798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065821    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-kube-api-access-qprgg\" (UniqueName: \"kubernetes.io/projected/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-kube-api-access-qprgg\") pod \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") "
Nov 03 08:24:28.066798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065870    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-workload-certs\") pod \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\" (UID: \"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d\") "
Nov 03 08:24:28.066798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065899    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-istio-envoy\") pod \"f813310f-254a-4b89-940c-06d7b335a112\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") "
Nov 03 08:24:28.066798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065904    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-credential-socket" pod "f813310f-254a-4b89-940c-06d7b335a112" (UID: "f813310f-254a-4b89-940c-06d7b335a112"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:28.066798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.065985    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-workload-certs\") pod \"f813310f-254a-4b89-940c-06d7b335a112\" (UID: \"f813310f-254a-4b89-940c-06d7b335a112\") "
Nov 03 08:24:28.066798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.066100    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-credential-socket" pod "0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" (UID: "0b4af5de-70dc-4fb5-b542-b4ccbd7c709d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:28.066798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.066129    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-data" pod "5e519992-15af-4d72-ba82-190c90ab0aa8" (UID: "5e519992-15af-4d72-ba82-190c90ab0aa8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:28.066798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.066121    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-podinfo\") pod \"5e519992-15af-4d72-ba82-190c90ab0aa8\" (UID: \"5e519992-15af-4d72-ba82-190c90ab0aa8\") "
Nov 03 08:24:28.066798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.066261    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-workload-socket" pod "5e519992-15af-4d72-ba82-190c90ab0aa8" (UID: "5e519992-15af-4d72-ba82-190c90ab0aa8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:28.066798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.066361    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-data" pod "0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" (UID: "0b4af5de-70dc-4fb5-b542-b4ccbd7c709d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:28.066798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.066385    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f813310f-254a-4b89-940c-06d7b335a112-istiod-ca-cert" pod "f813310f-254a-4b89-940c-06d7b335a112" (UID: "f813310f-254a-4b89-940c-06d7b335a112"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:28.066798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.066656    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-workload-certs" pod "5e519992-15af-4d72-ba82-190c90ab0aa8" (UID: "5e519992-15af-4d72-ba82-190c90ab0aa8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:28.067559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.066923    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f813310f-254a-4b89-940c-06d7b335a112-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.067559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.066950    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbhvx\" (UniqueName: \"kubernetes.io/projected/c3d60ea0-edd4-4c91-87be-9b414765dd7c-kube-api-access-zbhvx\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.067559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.066973    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.067559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.066989    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.067559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.067002    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.067559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.067016    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.067559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.067029    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.067559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.067043    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.067559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.067056    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.067559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.067069    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.067559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.067083    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.067559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.067096    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.067559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.067109    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.067559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.067122    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.067559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.067139    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.067559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.067152    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.067559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.067168    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.067559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.067182    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.067559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.067196    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3d60ea0-edd4-4c91-87be-9b414765dd7c-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.067559 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.067467    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-workload-certs" pod "f813310f-254a-4b89-940c-06d7b335a112" (UID: "f813310f-254a-4b89-940c-06d7b335a112"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:28.068683 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.067558    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e519992-15af-4d72-ba82-190c90ab0aa8-istiod-ca-cert" pod "5e519992-15af-4d72-ba82-190c90ab0aa8" (UID: "5e519992-15af-4d72-ba82-190c90ab0aa8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:28.068683 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.067656    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-workload-certs" pod "0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" (UID: "0b4af5de-70dc-4fb5-b542-b4ccbd7c709d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:28.068821 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.068686    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-podinfo" pod "5e519992-15af-4d72-ba82-190c90ab0aa8" (UID: "5e519992-15af-4d72-ba82-190c90ab0aa8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:28.069393 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.069361    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-token" pod "0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" (UID: "0b4af5de-70dc-4fb5-b542-b4ccbd7c709d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:28.069501 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.069427    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-envoy" pod "5e519992-15af-4d72-ba82-190c90ab0aa8" (UID: "5e519992-15af-4d72-ba82-190c90ab0aa8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:28.069575 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.069551    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-istio-envoy" pod "f813310f-254a-4b89-940c-06d7b335a112" (UID: "f813310f-254a-4b89-940c-06d7b335a112"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:28.069650 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.069596    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-token" pod "5e519992-15af-4d72-ba82-190c90ab0aa8" (UID: "5e519992-15af-4d72-ba82-190c90ab0aa8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:28.069996 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.069971    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-podinfo" pod "0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" (UID: "0b4af5de-70dc-4fb5-b542-b4ccbd7c709d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:28.070312 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.070282    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-kube-api-access-qprgg" pod "0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" (UID: "0b4af5de-70dc-4fb5-b542-b4ccbd7c709d"). InnerVolumeSpecName "kube-api-access-qprgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:28.070484 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.070466    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-envoy" pod "0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" (UID: "0b4af5de-70dc-4fb5-b542-b4ccbd7c709d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:28.070709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.070691    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e519992-15af-4d72-ba82-190c90ab0aa8-kube-api-access-gpx2h" pod "5e519992-15af-4d72-ba82-190c90ab0aa8" (UID: "5e519992-15af-4d72-ba82-190c90ab0aa8"). InnerVolumeSpecName "kube-api-access-gpx2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:28.070753 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.070718    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f813310f-254a-4b89-940c-06d7b335a112-istio-token" pod "f813310f-254a-4b89-940c-06d7b335a112" (UID: "f813310f-254a-4b89-940c-06d7b335a112"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:28.076042 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.076023    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f813310f-254a-4b89-940c-06d7b335a112-kube-api-access-jjxdm" pod "f813310f-254a-4b89-940c-06d7b335a112" (UID: "f813310f-254a-4b89-940c-06d7b335a112"). InnerVolumeSpecName "kube-api-access-jjxdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:28.087040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.087019    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f813310f-254a-4b89-940c-06d7b335a112-istio-podinfo" pod "f813310f-254a-4b89-940c-06d7b335a112" (UID: "f813310f-254a-4b89-940c-06d7b335a112"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:28.168995 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.168970    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.168995 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.168994    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.169211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.169016    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjxdm\" (UniqueName: \"kubernetes.io/projected/f813310f-254a-4b89-940c-06d7b335a112-kube-api-access-jjxdm\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.169211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.169034    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.169211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.169043    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.169211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.169052    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f813310f-254a-4b89-940c-06d7b335a112-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.169211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.169060    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f813310f-254a-4b89-940c-06d7b335a112-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.169211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.169069    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.169211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.169078    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.169211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.169087    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpx2h\" (UniqueName: \"kubernetes.io/projected/5e519992-15af-4d72-ba82-190c90ab0aa8-kube-api-access-gpx2h\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.169211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.169096    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e519992-15af-4d72-ba82-190c90ab0aa8-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.169211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.169105    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e519992-15af-4d72-ba82-190c90ab0aa8-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.169211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.169114    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qprgg\" (UniqueName: \"kubernetes.io/projected/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-kube-api-access-qprgg\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.169211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.169124    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.169211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.169133    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f813310f-254a-4b89-940c-06d7b335a112-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:28.218316 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.218283    5070 generic.go:356] "Generic (PLEG): container finished" podID="5e519992-15af-4d72-ba82-190c90ab0aa8" containerID="9c93261828483c8ee6e805de46f8a2dc36f1a94de5507e5444255362cf4b45f6" exitCode=0
Nov 03 08:24:28.218500 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.218331    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"
Nov 03 08:24:28.218500 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.218359    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm" event={"ID":"5e519992-15af-4d72-ba82-190c90ab0aa8","Type":"ContainerDied","Data":"9c93261828483c8ee6e805de46f8a2dc36f1a94de5507e5444255362cf4b45f6"}
Nov 03 08:24:28.218500 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.218412    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm" event={"ID":"5e519992-15af-4d72-ba82-190c90ab0aa8","Type":"ContainerDied","Data":"165f279145bf9a92f6d81d8c549c8fa3cff51c6cd985b46f9b80d8a816000313"}
Nov 03 08:24:28.218500 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.218440    5070 scope.go:122] "RemoveContainer" containerID="9c93261828483c8ee6e805de46f8a2dc36f1a94de5507e5444255362cf4b45f6"
Nov 03 08:24:28.219960 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.219932    5070 generic.go:356] "Generic (PLEG): container finished" podID="f813310f-254a-4b89-940c-06d7b335a112" containerID="d67b2683a3325f8a6b25d5aacdac5136a40650214806444b57b9c707da90d03f" exitCode=0
Nov 03 08:24:28.220109 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.219977    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"
Nov 03 08:24:28.220109 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.220010    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929" event={"ID":"f813310f-254a-4b89-940c-06d7b335a112","Type":"ContainerDied","Data":"d67b2683a3325f8a6b25d5aacdac5136a40650214806444b57b9c707da90d03f"}
Nov 03 08:24:28.220109 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.220065    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929" event={"ID":"f813310f-254a-4b89-940c-06d7b335a112","Type":"ContainerDied","Data":"b4164b1f8cfa999d9d0292ef5b14aefce05b295d0fd3d8709df355e6c9973311"}
Nov 03 08:24:28.221822 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.221797    5070 generic.go:356] "Generic (PLEG): container finished" podID="0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" containerID="c0dac5751a2dd2612c2e5ab4921f8f6c0fe90caa76eeee71bb742ad563d7e24a" exitCode=0
Nov 03 08:24:28.221939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.221829    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"
Nov 03 08:24:28.222009 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.221995    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm" event={"ID":"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d","Type":"ContainerDied","Data":"c0dac5751a2dd2612c2e5ab4921f8f6c0fe90caa76eeee71bb742ad563d7e24a"}
Nov 03 08:24:28.222062 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.222037    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm" event={"ID":"0b4af5de-70dc-4fb5-b542-b4ccbd7c709d","Type":"ContainerDied","Data":"0c17f4eebf184b37cad44175c7f8629da014f07ac29bb75c09cc9b64bd3cc916"}
Nov 03 08:24:28.224479 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.224454    5070 generic.go:356] "Generic (PLEG): container finished" podID="c3d60ea0-edd4-4c91-87be-9b414765dd7c" containerID="95c97a02c5b2b0d10a4ad46e0a3d7ad73a3e1a56ccb837840cf4f6983beea19e" exitCode=0
Nov 03 08:24:28.224587 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.224493    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd" event={"ID":"c3d60ea0-edd4-4c91-87be-9b414765dd7c","Type":"ContainerDied","Data":"95c97a02c5b2b0d10a4ad46e0a3d7ad73a3e1a56ccb837840cf4f6983beea19e"}
Nov 03 08:24:28.224587 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.224530    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd" event={"ID":"c3d60ea0-edd4-4c91-87be-9b414765dd7c","Type":"ContainerDied","Data":"b4f1383a3f80bef9c2b8ad3445ee25eb447e5b333eb0591408058397a714229a"}
Nov 03 08:24:28.224697 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.224595    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"
Nov 03 08:24:28.233077 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.232843    5070 scope.go:122] "RemoveContainer" containerID="409bb03d7b86506848bbbbc07a8c1b45dc59431ec30d50f76ee16014d3da0711"
Nov 03 08:24:28.246357 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.246336    5070 scope.go:122] "RemoveContainer" containerID="9c93261828483c8ee6e805de46f8a2dc36f1a94de5507e5444255362cf4b45f6"
Nov 03 08:24:28.246636 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:28.246611    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c93261828483c8ee6e805de46f8a2dc36f1a94de5507e5444255362cf4b45f6\": container with ID starting with 9c93261828483c8ee6e805de46f8a2dc36f1a94de5507e5444255362cf4b45f6 not found: ID does not exist" containerID="9c93261828483c8ee6e805de46f8a2dc36f1a94de5507e5444255362cf4b45f6"
Nov 03 08:24:28.246691 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.246646    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c93261828483c8ee6e805de46f8a2dc36f1a94de5507e5444255362cf4b45f6"} err="failed to get container status \"9c93261828483c8ee6e805de46f8a2dc36f1a94de5507e5444255362cf4b45f6\": rpc error: code = NotFound desc = could not find container \"9c93261828483c8ee6e805de46f8a2dc36f1a94de5507e5444255362cf4b45f6\": container with ID starting with 9c93261828483c8ee6e805de46f8a2dc36f1a94de5507e5444255362cf4b45f6 not found: ID does not exist"
Nov 03 08:24:28.246691 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.246670    5070 scope.go:122] "RemoveContainer" containerID="409bb03d7b86506848bbbbc07a8c1b45dc59431ec30d50f76ee16014d3da0711"
Nov 03 08:24:28.246969 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:28.246948    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"409bb03d7b86506848bbbbc07a8c1b45dc59431ec30d50f76ee16014d3da0711\": container with ID starting with 409bb03d7b86506848bbbbc07a8c1b45dc59431ec30d50f76ee16014d3da0711 not found: ID does not exist" containerID="409bb03d7b86506848bbbbc07a8c1b45dc59431ec30d50f76ee16014d3da0711"
Nov 03 08:24:28.247019 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.246981    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"]
Nov 03 08:24:28.247019 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.246977    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"409bb03d7b86506848bbbbc07a8c1b45dc59431ec30d50f76ee16014d3da0711"} err="failed to get container status \"409bb03d7b86506848bbbbc07a8c1b45dc59431ec30d50f76ee16014d3da0711\": rpc error: code = NotFound desc = could not find container \"409bb03d7b86506848bbbbc07a8c1b45dc59431ec30d50f76ee16014d3da0711\": container with ID starting with 409bb03d7b86506848bbbbc07a8c1b45dc59431ec30d50f76ee16014d3da0711 not found: ID does not exist"
Nov 03 08:24:28.247019 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.247016    5070 scope.go:122] "RemoveContainer" containerID="d67b2683a3325f8a6b25d5aacdac5136a40650214806444b57b9c707da90d03f"
Nov 03 08:24:28.251147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.251126    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-nj6gd"]
Nov 03 08:24:28.257736 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.257680    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"]
Nov 03 08:24:28.259547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.259528    5070 scope.go:122] "RemoveContainer" containerID="d67b2683a3325f8a6b25d5aacdac5136a40650214806444b57b9c707da90d03f"
Nov 03 08:24:28.259874 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:28.259843    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d67b2683a3325f8a6b25d5aacdac5136a40650214806444b57b9c707da90d03f\": container with ID starting with d67b2683a3325f8a6b25d5aacdac5136a40650214806444b57b9c707da90d03f not found: ID does not exist" containerID="d67b2683a3325f8a6b25d5aacdac5136a40650214806444b57b9c707da90d03f"
Nov 03 08:24:28.259974 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.259880    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d67b2683a3325f8a6b25d5aacdac5136a40650214806444b57b9c707da90d03f"} err="failed to get container status \"d67b2683a3325f8a6b25d5aacdac5136a40650214806444b57b9c707da90d03f\": rpc error: code = NotFound desc = could not find container \"d67b2683a3325f8a6b25d5aacdac5136a40650214806444b57b9c707da90d03f\": container with ID starting with d67b2683a3325f8a6b25d5aacdac5136a40650214806444b57b9c707da90d03f not found: ID does not exist"
Nov 03 08:24:28.259974 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.259902    5070 scope.go:122] "RemoveContainer" containerID="c0dac5751a2dd2612c2e5ab4921f8f6c0fe90caa76eeee71bb742ad563d7e24a"
Nov 03 08:24:28.262642 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.262624    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-dbgjm"]
Nov 03 08:24:28.271302 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.271274    5070 scope.go:122] "RemoveContainer" containerID="c0dac5751a2dd2612c2e5ab4921f8f6c0fe90caa76eeee71bb742ad563d7e24a"
Nov 03 08:24:28.271593 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:28.271575    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0dac5751a2dd2612c2e5ab4921f8f6c0fe90caa76eeee71bb742ad563d7e24a\": container with ID starting with c0dac5751a2dd2612c2e5ab4921f8f6c0fe90caa76eeee71bb742ad563d7e24a not found: ID does not exist" containerID="c0dac5751a2dd2612c2e5ab4921f8f6c0fe90caa76eeee71bb742ad563d7e24a"
Nov 03 08:24:28.271677 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.271598    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0dac5751a2dd2612c2e5ab4921f8f6c0fe90caa76eeee71bb742ad563d7e24a"} err="failed to get container status \"c0dac5751a2dd2612c2e5ab4921f8f6c0fe90caa76eeee71bb742ad563d7e24a\": rpc error: code = NotFound desc = could not find container \"c0dac5751a2dd2612c2e5ab4921f8f6c0fe90caa76eeee71bb742ad563d7e24a\": container with ID starting with c0dac5751a2dd2612c2e5ab4921f8f6c0fe90caa76eeee71bb742ad563d7e24a not found: ID does not exist"
Nov 03 08:24:28.271677 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.271614    5070 scope.go:122] "RemoveContainer" containerID="95c97a02c5b2b0d10a4ad46e0a3d7ad73a3e1a56ccb837840cf4f6983beea19e"
Nov 03 08:24:28.288825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.280829    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"]
Nov 03 08:24:28.288825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.286971    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-hztvm"]
Nov 03 08:24:28.291920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.291223    5070 scope.go:122] "RemoveContainer" containerID="95c97a02c5b2b0d10a4ad46e0a3d7ad73a3e1a56ccb837840cf4f6983beea19e"
Nov 03 08:24:28.291920 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:28.291607    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c97a02c5b2b0d10a4ad46e0a3d7ad73a3e1a56ccb837840cf4f6983beea19e\": container with ID starting with 95c97a02c5b2b0d10a4ad46e0a3d7ad73a3e1a56ccb837840cf4f6983beea19e not found: ID does not exist" containerID="95c97a02c5b2b0d10a4ad46e0a3d7ad73a3e1a56ccb837840cf4f6983beea19e"
Nov 03 08:24:28.291920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.291640    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c97a02c5b2b0d10a4ad46e0a3d7ad73a3e1a56ccb837840cf4f6983beea19e"} err="failed to get container status \"95c97a02c5b2b0d10a4ad46e0a3d7ad73a3e1a56ccb837840cf4f6983beea19e\": rpc error: code = NotFound desc = could not find container \"95c97a02c5b2b0d10a4ad46e0a3d7ad73a3e1a56ccb837840cf4f6983beea19e\": container with ID starting with 95c97a02c5b2b0d10a4ad46e0a3d7ad73a3e1a56ccb837840cf4f6983beea19e not found: ID does not exist"
Nov 03 08:24:28.293059 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.293037    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"]
Nov 03 08:24:28.298178 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:28.298150    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-616e46-openshift-default-75d55d5cc-rr929"]
Nov 03 08:24:30.026148 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:30.026117    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" path="/var/lib/kubelet/pods/0b4af5de-70dc-4fb5-b542-b4ccbd7c709d/volumes"
Nov 03 08:24:30.026598 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:30.026584    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e519992-15af-4d72-ba82-190c90ab0aa8" path="/var/lib/kubelet/pods/5e519992-15af-4d72-ba82-190c90ab0aa8/volumes"
Nov 03 08:24:30.027065 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:30.027051    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3d60ea0-edd4-4c91-87be-9b414765dd7c" path="/var/lib/kubelet/pods/c3d60ea0-edd4-4c91-87be-9b414765dd7c/volumes"
Nov 03 08:24:30.027833 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:30.027816    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f813310f-254a-4b89-940c-06d7b335a112" path="/var/lib/kubelet/pods/f813310f-254a-4b89-940c-06d7b335a112/volumes"
Nov 03 08:24:32.524225 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:32.524172    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p"]
Nov 03 08:24:32.530957 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:32.530904    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b8a152-scp4p"]
Nov 03 08:24:33.285985 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:33.285951    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd"]
Nov 03 08:24:34.027148 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:34.027114    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="803348c9-5802-4b5a-836d-462fb6ebc404" path="/var/lib/kubelet/pods/803348c9-5802-4b5a-836d-462fb6ebc404/volumes"
Nov 03 08:24:37.863080 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.863052    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8"]
Nov 03 08:24:37.863721 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.863705    5070 cpu_manager.go:414] "containerMap: removing container" podUID="f813310f-254a-4b89-940c-06d7b335a112" containerName="istio-proxy"
Nov 03 08:24:37.863721 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.863721    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f813310f-254a-4b89-940c-06d7b335a112" containerName="istio-proxy"
Nov 03 08:24:37.863824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.863735    5070 cpu_manager.go:414] "containerMap: removing container" podUID="0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" containerName="istio-proxy"
Nov 03 08:24:37.863824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.863741    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" containerName="istio-proxy"
Nov 03 08:24:37.863824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.863772    5070 cpu_manager.go:414] "containerMap: removing container" podUID="c3d60ea0-edd4-4c91-87be-9b414765dd7c" containerName="istio-proxy"
Nov 03 08:24:37.863824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.863778    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d60ea0-edd4-4c91-87be-9b414765dd7c" containerName="istio-proxy"
Nov 03 08:24:37.863824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.863785    5070 cpu_manager.go:414] "containerMap: removing container" podUID="2484ae5d-667e-4ee2-a1fa-01c61ded3d45" containerName="revoke-keys"
Nov 03 08:24:37.863824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.863789    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2484ae5d-667e-4ee2-a1fa-01c61ded3d45" containerName="revoke-keys"
Nov 03 08:24:37.863824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.863805    5070 cpu_manager.go:414] "containerMap: removing container" podUID="5e519992-15af-4d72-ba82-190c90ab0aa8" containerName="istio-proxy"
Nov 03 08:24:37.863824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.863810    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e519992-15af-4d72-ba82-190c90ab0aa8" containerName="istio-proxy"
Nov 03 08:24:37.863824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.863824    5070 cpu_manager.go:414] "containerMap: removing container" podUID="5e519992-15af-4d72-ba82-190c90ab0aa8" containerName="istio-proxy"
Nov 03 08:24:37.864093 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.863832    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e519992-15af-4d72-ba82-190c90ab0aa8" containerName="istio-proxy"
Nov 03 08:24:37.864093 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.863947    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="f813310f-254a-4b89-940c-06d7b335a112" containerName="istio-proxy"
Nov 03 08:24:37.864093 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.863954    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b4af5de-70dc-4fb5-b542-b4ccbd7c709d" containerName="istio-proxy"
Nov 03 08:24:37.864093 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.863965    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d60ea0-edd4-4c91-87be-9b414765dd7c" containerName="istio-proxy"
Nov 03 08:24:37.864093 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.863970    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="2484ae5d-667e-4ee2-a1fa-01c61ded3d45" containerName="revoke-keys"
Nov 03 08:24:37.864093 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.863977    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e519992-15af-4d72-ba82-190c90ab0aa8" containerName="istio-proxy"
Nov 03 08:24:37.864093 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.863982    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e519992-15af-4d72-ba82-190c90ab0aa8" containerName="istio-proxy"
Nov 03 08:24:37.869394 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.869375    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8"
Nov 03 08:24:37.872343 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.872317    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:24:37.872343 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.872332    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:24:37.872610 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.872319    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:24:37.872610 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.872407    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:24:37.876296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.876270    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5"]
Nov 03 08:24:37.882674 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.882402    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8"]
Nov 03 08:24:37.882674 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.882510    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5"
Nov 03 08:24:37.890826 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.890801    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5"]
Nov 03 08:24:37.977687 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.977660    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c070983-a6e2-41ce-aff5-8526f7d99375-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a14a1e4-jpxs8\" (UID: \"0c070983-a6e2-41ce-aff5-8526f7d99375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8"
Nov 03 08:24:37.977845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.977809    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0820001-c7d2-49f8-afa6-79038e19c795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a06b773-drqz5\" (UID: \"d0820001-c7d2-49f8-afa6-79038e19c795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5"
Nov 03 08:24:37.977845 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.977837    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg486\" (UniqueName: \"kubernetes.io/projected/0c070983-a6e2-41ce-aff5-8526f7d99375-kube-api-access-tg486\") pod \"maas-api-revoke-sub-3a14a1e4-jpxs8\" (UID: \"0c070983-a6e2-41ce-aff5-8526f7d99375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8"
Nov 03 08:24:37.977950 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.977884    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c070983-a6e2-41ce-aff5-8526f7d99375-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a14a1e4-jpxs8\" (UID: \"0c070983-a6e2-41ce-aff5-8526f7d99375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8"
Nov 03 08:24:37.977987 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.977955    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg5nc\" (UniqueName: \"kubernetes.io/projected/d0820001-c7d2-49f8-afa6-79038e19c795-kube-api-access-cg5nc\") pod \"maas-api-revoke-sub-0a06b773-drqz5\" (UID: \"d0820001-c7d2-49f8-afa6-79038e19c795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5"
Nov 03 08:24:37.978032 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:37.977996    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0820001-c7d2-49f8-afa6-79038e19c795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a06b773-drqz5\" (UID: \"d0820001-c7d2-49f8-afa6-79038e19c795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5"
Nov 03 08:24:38.079250 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:38.079224    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c070983-a6e2-41ce-aff5-8526f7d99375-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a14a1e4-jpxs8\" (UID: \"0c070983-a6e2-41ce-aff5-8526f7d99375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8"
Nov 03 08:24:38.079405 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:38.079320    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0820001-c7d2-49f8-afa6-79038e19c795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a06b773-drqz5\" (UID: \"d0820001-c7d2-49f8-afa6-79038e19c795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5"
Nov 03 08:24:38.079405 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:38.079351    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg486\" (UniqueName: \"kubernetes.io/projected/0c070983-a6e2-41ce-aff5-8526f7d99375-kube-api-access-tg486\") pod \"maas-api-revoke-sub-3a14a1e4-jpxs8\" (UID: \"0c070983-a6e2-41ce-aff5-8526f7d99375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8"
Nov 03 08:24:38.079405 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:38.079376    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c070983-a6e2-41ce-aff5-8526f7d99375-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a14a1e4-jpxs8\" (UID: \"0c070983-a6e2-41ce-aff5-8526f7d99375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8"
Nov 03 08:24:38.079579 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:38.079435    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg5nc\" (UniqueName: \"kubernetes.io/projected/d0820001-c7d2-49f8-afa6-79038e19c795-kube-api-access-cg5nc\") pod \"maas-api-revoke-sub-0a06b773-drqz5\" (UID: \"d0820001-c7d2-49f8-afa6-79038e19c795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5"
Nov 03 08:24:38.079579 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:38.079483    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0820001-c7d2-49f8-afa6-79038e19c795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a06b773-drqz5\" (UID: \"d0820001-c7d2-49f8-afa6-79038e19c795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5"
Nov 03 08:24:38.080066 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:38.080045    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c070983-a6e2-41ce-aff5-8526f7d99375-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a14a1e4-jpxs8\" (UID: \"0c070983-a6e2-41ce-aff5-8526f7d99375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8"
Nov 03 08:24:38.080305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:38.080279    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0820001-c7d2-49f8-afa6-79038e19c795-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a06b773-drqz5\" (UID: \"d0820001-c7d2-49f8-afa6-79038e19c795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5"
Nov 03 08:24:38.081993 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:38.081967    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0820001-c7d2-49f8-afa6-79038e19c795-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a06b773-drqz5\" (UID: \"d0820001-c7d2-49f8-afa6-79038e19c795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5"
Nov 03 08:24:38.082130 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:38.082114    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c070983-a6e2-41ce-aff5-8526f7d99375-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a14a1e4-jpxs8\" (UID: \"0c070983-a6e2-41ce-aff5-8526f7d99375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8"
Nov 03 08:24:38.090484 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:38.090461    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg486\" (UniqueName: \"kubernetes.io/projected/0c070983-a6e2-41ce-aff5-8526f7d99375-kube-api-access-tg486\") pod \"maas-api-revoke-sub-3a14a1e4-jpxs8\" (UID: \"0c070983-a6e2-41ce-aff5-8526f7d99375\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8"
Nov 03 08:24:38.090617 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:38.090599    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg5nc\" (UniqueName: \"kubernetes.io/projected/d0820001-c7d2-49f8-afa6-79038e19c795-kube-api-access-cg5nc\") pod \"maas-api-revoke-sub-0a06b773-drqz5\" (UID: \"d0820001-c7d2-49f8-afa6-79038e19c795\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5"
Nov 03 08:24:38.187938 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:38.187881    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8"
Nov 03 08:24:38.201421 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:38.201400    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5"
Nov 03 08:24:38.310318 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:38.309753    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd" podUID="cb961482-e401-47f5-8aff-f604b364d631" containerName="payload-processing" containerID="cri-o://493c820308728c3e98ad907864a52d22363893328c3b730567fe658c759f2a07" gracePeriod=25
Nov 03 08:24:38.479675 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:38.479647    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8"]
Nov 03 08:24:38.480287 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:24:38.480262    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c070983_a6e2_41ce_aff5_8526f7d99375.slice/crio-4c8a0c2a3d7dc23a8247b82ac008455330d05ad6298aa38fa546d31aa34565eb WatchSource:0}: Error finding container 4c8a0c2a3d7dc23a8247b82ac008455330d05ad6298aa38fa546d31aa34565eb: Status 404 returned error can't find the container with id 4c8a0c2a3d7dc23a8247b82ac008455330d05ad6298aa38fa546d31aa34565eb
Nov 03 08:24:38.547074 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:24:38.547047    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0820001_c7d2_49f8_afa6_79038e19c795.slice/crio-4d0b2ba66b69584cb1e2fd6fc92563dc00fbd496717ca507df8e440c4d7cd66b WatchSource:0}: Error finding container 4d0b2ba66b69584cb1e2fd6fc92563dc00fbd496717ca507df8e440c4d7cd66b: Status 404 returned error can't find the container with id 4d0b2ba66b69584cb1e2fd6fc92563dc00fbd496717ca507df8e440c4d7cd66b
Nov 03 08:24:38.547619 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:38.547578    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5"]
Nov 03 08:24:39.095258 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.095142    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"]
Nov 03 08:24:39.095717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.095407    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx" podUID="a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd" containerName="istio-proxy" containerID="cri-o://a9cf4d155950d607fdf623e6e220b88a4bc525f0c17872ffcc38eea4eca176c3" gracePeriod=30
Nov 03 08:24:39.102403 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.101024    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"]
Nov 03 08:24:39.102403 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.101230    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb" podUID="d915ac09-a22a-4e5d-943a-97cdeb2d9c24" containerName="istio-proxy" containerID="cri-o://e07f6abee64d466bfa79ecc820f3501f1b879c5f0b289677d99f154f78040f1a" gracePeriod=30
Nov 03 08:24:39.109983 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.108572    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"]
Nov 03 08:24:39.109983 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.108854    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b" podUID="825e9af0-37a8-4d27-9a34-20b2f87e92d5" containerName="istio-proxy" containerID="cri-o://52967583fe3b748836a41c5cc2f924344cc62b23ae21445430993d8895d83eb2" gracePeriod=30
Nov 03 08:24:39.115204 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.115085    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"]
Nov 03 08:24:39.115315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.115287    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz" podUID="d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" containerName="istio-proxy" containerID="cri-o://e222ed448d5eb626b27ddaf01aa7c0996e4e51c20eed16e3fa24ac4c005a5768" gracePeriod=30
Nov 03 08:24:39.122088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.122032    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"]
Nov 03 08:24:39.122450 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.122416    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn" podUID="8fab335e-1024-4086-92ef-49b496f0abc0" containerName="istio-proxy" containerID="cri-o://56b5fd507f150bc77dc9b23955016dd8d479ed5d3d9673f1c4d686954d0c2f30" gracePeriod=30
Nov 03 08:24:39.375975 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.375933    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8" event={"ID":"0c070983-a6e2-41ce-aff5-8526f7d99375","Type":"ContainerStarted","Data":"4fc053c261ba3813371ae3b9132f359d0ba3c6133718d9eac2ecd7cae39663b8"}
Nov 03 08:24:39.376171 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.376026    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8" event={"ID":"0c070983-a6e2-41ce-aff5-8526f7d99375","Type":"ContainerStarted","Data":"4c8a0c2a3d7dc23a8247b82ac008455330d05ad6298aa38fa546d31aa34565eb"}
Nov 03 08:24:39.377753 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.377726    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5" event={"ID":"d0820001-c7d2-49f8-afa6-79038e19c795","Type":"ContainerStarted","Data":"0d99ce8d05353067d19e91966595d6585d95d9bf9244328113c135e5a14b83c6"}
Nov 03 08:24:39.377909 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.377814    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5" event={"ID":"d0820001-c7d2-49f8-afa6-79038e19c795","Type":"ContainerStarted","Data":"4d0b2ba66b69584cb1e2fd6fc92563dc00fbd496717ca507df8e440c4d7cd66b"}
Nov 03 08:24:39.391363 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.391294    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8" podStartSLOduration=2.391283429 podStartE2EDuration="2.391283429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:24:39.390689483 +0000 UTC m=+1877.505153413" watchObservedRunningTime="2026-11-03 08:24:39.391283429 +0000 UTC m=+1877.505747351"
Nov 03 08:24:39.409587 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.409550    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5" podStartSLOduration=2.409538788 podStartE2EDuration="2.409538788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:24:39.40528106 +0000 UTC m=+1877.519744991" watchObservedRunningTime="2026-11-03 08:24:39.409538788 +0000 UTC m=+1877.524002728"
Nov 03 08:24:39.843308 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.843277    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4"]
Nov 03 08:24:39.847706 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.847683    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4"
Nov 03 08:24:39.856454 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.856307    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4"]
Nov 03 08:24:39.949630 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.949555    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb66350-fefe-4b07-bd2c-f633a0bbfee3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4\" (UID: \"bfb66350-fefe-4b07-bd2c-f633a0bbfee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4"
Nov 03 08:24:39.949630 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.949604    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb66350-fefe-4b07-bd2c-f633a0bbfee3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4\" (UID: \"bfb66350-fefe-4b07-bd2c-f633a0bbfee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4"
Nov 03 08:24:39.949883 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:39.949695    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt5mz\" (UniqueName: \"kubernetes.io/projected/bfb66350-fefe-4b07-bd2c-f633a0bbfee3-kube-api-access-xt5mz\") pod \"maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4\" (UID: \"bfb66350-fefe-4b07-bd2c-f633a0bbfee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4"
Nov 03 08:24:40.051374 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:40.051263    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb66350-fefe-4b07-bd2c-f633a0bbfee3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4\" (UID: \"bfb66350-fefe-4b07-bd2c-f633a0bbfee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4"
Nov 03 08:24:40.051374 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:40.051368    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb66350-fefe-4b07-bd2c-f633a0bbfee3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4\" (UID: \"bfb66350-fefe-4b07-bd2c-f633a0bbfee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4"
Nov 03 08:24:40.051564 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:40.051432    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt5mz\" (UniqueName: \"kubernetes.io/projected/bfb66350-fefe-4b07-bd2c-f633a0bbfee3-kube-api-access-xt5mz\") pod \"maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4\" (UID: \"bfb66350-fefe-4b07-bd2c-f633a0bbfee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4"
Nov 03 08:24:40.052169 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:40.052144    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb66350-fefe-4b07-bd2c-f633a0bbfee3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4\" (UID: \"bfb66350-fefe-4b07-bd2c-f633a0bbfee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4"
Nov 03 08:24:40.053796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:40.053769    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb66350-fefe-4b07-bd2c-f633a0bbfee3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4\" (UID: \"bfb66350-fefe-4b07-bd2c-f633a0bbfee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4"
Nov 03 08:24:40.063356 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:40.063325    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt5mz\" (UniqueName: \"kubernetes.io/projected/bfb66350-fefe-4b07-bd2c-f633a0bbfee3-kube-api-access-xt5mz\") pod \"maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4\" (UID: \"bfb66350-fefe-4b07-bd2c-f633a0bbfee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4"
Nov 03 08:24:40.165152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:40.164476    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4"
Nov 03 08:24:40.389296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:40.389263    5070 generic.go:356] "Generic (PLEG): container finished" podID="0c070983-a6e2-41ce-aff5-8526f7d99375" containerID="4fc053c261ba3813371ae3b9132f359d0ba3c6133718d9eac2ecd7cae39663b8" exitCode=0
Nov 03 08:24:40.389457 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:40.389333    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8" event={"ID":"0c070983-a6e2-41ce-aff5-8526f7d99375","Type":"ContainerDied","Data":"4fc053c261ba3813371ae3b9132f359d0ba3c6133718d9eac2ecd7cae39663b8"}
Nov 03 08:24:40.391287 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:40.391261    5070 generic.go:356] "Generic (PLEG): container finished" podID="d0820001-c7d2-49f8-afa6-79038e19c795" containerID="0d99ce8d05353067d19e91966595d6585d95d9bf9244328113c135e5a14b83c6" exitCode=0
Nov 03 08:24:40.391456 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:40.391352    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5" event={"ID":"d0820001-c7d2-49f8-afa6-79038e19c795","Type":"ContainerDied","Data":"0d99ce8d05353067d19e91966595d6585d95d9bf9244328113c135e5a14b83c6"}
Nov 03 08:24:40.469059 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:40.469022    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4"]
Nov 03 08:24:40.473541 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:24:40.473512    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb66350_fefe_4b07_bd2c_f633a0bbfee3.slice/crio-e264c83e5d994a8d5b468b895b7071839c0eea09b930b69a1e1e20febc91bba7 WatchSource:0}: Error finding container e264c83e5d994a8d5b468b895b7071839c0eea09b930b69a1e1e20febc91bba7: Status 404 returned error can't find the container with id e264c83e5d994a8d5b468b895b7071839c0eea09b930b69a1e1e20febc91bba7
Nov 03 08:24:41.403836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.403793    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4" event={"ID":"bfb66350-fefe-4b07-bd2c-f633a0bbfee3","Type":"ContainerStarted","Data":"2397148c59a145e68d69f487d17e256858d0f22cff879614978964f940aa4219"}
Nov 03 08:24:41.404313 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.403865    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4" event={"ID":"bfb66350-fefe-4b07-bd2c-f633a0bbfee3","Type":"ContainerStarted","Data":"e264c83e5d994a8d5b468b895b7071839c0eea09b930b69a1e1e20febc91bba7"}
Nov 03 08:24:41.425777 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.425678    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4" podStartSLOduration=2.425360177 podStartE2EDuration="2.425360177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:24:41.418160702 +0000 UTC m=+1879.532624634" watchObservedRunningTime="2026-11-03 08:24:41.425360177 +0000 UTC m=+1879.539824107"
Nov 03 08:24:41.657673 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.657613    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8"
Nov 03 08:24:41.661980 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.661950    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5"
Nov 03 08:24:41.786383 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.786351    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c070983-a6e2-41ce-aff5-8526f7d99375-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c070983-a6e2-41ce-aff5-8526f7d99375-maas-api-openshift-ca\") pod \"0c070983-a6e2-41ce-aff5-8526f7d99375\" (UID: \"0c070983-a6e2-41ce-aff5-8526f7d99375\") "
Nov 03 08:24:41.786383 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.786389    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c070983-a6e2-41ce-aff5-8526f7d99375-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c070983-a6e2-41ce-aff5-8526f7d99375-maas-api-xks-ca\") pod \"0c070983-a6e2-41ce-aff5-8526f7d99375\" (UID: \"0c070983-a6e2-41ce-aff5-8526f7d99375\") "
Nov 03 08:24:41.786611 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.786420    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0820001-c7d2-49f8-afa6-79038e19c795-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0820001-c7d2-49f8-afa6-79038e19c795-maas-api-openshift-ca\") pod \"d0820001-c7d2-49f8-afa6-79038e19c795\" (UID: \"d0820001-c7d2-49f8-afa6-79038e19c795\") "
Nov 03 08:24:41.786611 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.786488    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0820001-c7d2-49f8-afa6-79038e19c795-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0820001-c7d2-49f8-afa6-79038e19c795-maas-api-xks-ca\") pod \"d0820001-c7d2-49f8-afa6-79038e19c795\" (UID: \"d0820001-c7d2-49f8-afa6-79038e19c795\") "
Nov 03 08:24:41.786611 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.786574    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0820001-c7d2-49f8-afa6-79038e19c795-kube-api-access-cg5nc\" (UniqueName: \"kubernetes.io/projected/d0820001-c7d2-49f8-afa6-79038e19c795-kube-api-access-cg5nc\") pod \"d0820001-c7d2-49f8-afa6-79038e19c795\" (UID: \"d0820001-c7d2-49f8-afa6-79038e19c795\") "
Nov 03 08:24:41.786804 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.786660    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c070983-a6e2-41ce-aff5-8526f7d99375-kube-api-access-tg486\" (UniqueName: \"kubernetes.io/projected/0c070983-a6e2-41ce-aff5-8526f7d99375-kube-api-access-tg486\") pod \"0c070983-a6e2-41ce-aff5-8526f7d99375\" (UID: \"0c070983-a6e2-41ce-aff5-8526f7d99375\") "
Nov 03 08:24:41.786804 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.786748    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c070983-a6e2-41ce-aff5-8526f7d99375-maas-api-openshift-ca" pod "0c070983-a6e2-41ce-aff5-8526f7d99375" (UID: "0c070983-a6e2-41ce-aff5-8526f7d99375"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:41.786928 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.786844    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0820001-c7d2-49f8-afa6-79038e19c795-maas-api-openshift-ca" pod "d0820001-c7d2-49f8-afa6-79038e19c795" (UID: "d0820001-c7d2-49f8-afa6-79038e19c795"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:41.787532 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.787502    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c070983-a6e2-41ce-aff5-8526f7d99375-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:41.787678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.787533    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0820001-c7d2-49f8-afa6-79038e19c795-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:41.789034 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.789002    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c070983-a6e2-41ce-aff5-8526f7d99375-kube-api-access-tg486" pod "0c070983-a6e2-41ce-aff5-8526f7d99375" (UID: "0c070983-a6e2-41ce-aff5-8526f7d99375"). InnerVolumeSpecName "kube-api-access-tg486". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:41.789137 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.789042    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0820001-c7d2-49f8-afa6-79038e19c795-kube-api-access-cg5nc" pod "d0820001-c7d2-49f8-afa6-79038e19c795" (UID: "d0820001-c7d2-49f8-afa6-79038e19c795"). InnerVolumeSpecName "kube-api-access-cg5nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:41.789233 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.789213    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0820001-c7d2-49f8-afa6-79038e19c795-maas-api-xks-ca" pod "d0820001-c7d2-49f8-afa6-79038e19c795" (UID: "d0820001-c7d2-49f8-afa6-79038e19c795"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:24:41.789233 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.789227    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c070983-a6e2-41ce-aff5-8526f7d99375-maas-api-xks-ca" pod "0c070983-a6e2-41ce-aff5-8526f7d99375" (UID: "0c070983-a6e2-41ce-aff5-8526f7d99375"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:24:41.891838 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.890592    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c070983-a6e2-41ce-aff5-8526f7d99375-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:41.891838 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.890631    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0820001-c7d2-49f8-afa6-79038e19c795-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:41.891838 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.890652    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg5nc\" (UniqueName: \"kubernetes.io/projected/d0820001-c7d2-49f8-afa6-79038e19c795-kube-api-access-cg5nc\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:41.891838 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:41.890667    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg486\" (UniqueName: \"kubernetes.io/projected/0c070983-a6e2-41ce-aff5-8526f7d99375-kube-api-access-tg486\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:42.413604 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:42.413578    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5"
Nov 03 08:24:42.414033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:42.413603    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5" event={"ID":"d0820001-c7d2-49f8-afa6-79038e19c795","Type":"ContainerDied","Data":"4d0b2ba66b69584cb1e2fd6fc92563dc00fbd496717ca507df8e440c4d7cd66b"}
Nov 03 08:24:42.414033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:42.413677    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d0b2ba66b69584cb1e2fd6fc92563dc00fbd496717ca507df8e440c4d7cd66b"
Nov 03 08:24:42.415486 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:42.415457    5070 generic.go:356] "Generic (PLEG): container finished" podID="bfb66350-fefe-4b07-bd2c-f633a0bbfee3" containerID="2397148c59a145e68d69f487d17e256858d0f22cff879614978964f940aa4219" exitCode=0
Nov 03 08:24:42.415631 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:42.415538    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4" event={"ID":"bfb66350-fefe-4b07-bd2c-f633a0bbfee3","Type":"ContainerDied","Data":"2397148c59a145e68d69f487d17e256858d0f22cff879614978964f940aa4219"}
Nov 03 08:24:42.417169 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:42.417149    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8" event={"ID":"0c070983-a6e2-41ce-aff5-8526f7d99375","Type":"ContainerDied","Data":"4c8a0c2a3d7dc23a8247b82ac008455330d05ad6298aa38fa546d31aa34565eb"}
Nov 03 08:24:42.417291 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:42.417179    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c8a0c2a3d7dc23a8247b82ac008455330d05ad6298aa38fa546d31aa34565eb"
Nov 03 08:24:42.417291 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:42.417187    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8"
Nov 03 08:24:43.656413 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:43.656384    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4"
Nov 03 08:24:43.820851 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:43.820745    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfb66350-fefe-4b07-bd2c-f633a0bbfee3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb66350-fefe-4b07-bd2c-f633a0bbfee3-maas-api-openshift-ca\") pod \"bfb66350-fefe-4b07-bd2c-f633a0bbfee3\" (UID: \"bfb66350-fefe-4b07-bd2c-f633a0bbfee3\") "
Nov 03 08:24:43.821001 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:43.820854    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfb66350-fefe-4b07-bd2c-f633a0bbfee3-kube-api-access-xt5mz\" (UniqueName: \"kubernetes.io/projected/bfb66350-fefe-4b07-bd2c-f633a0bbfee3-kube-api-access-xt5mz\") pod \"bfb66350-fefe-4b07-bd2c-f633a0bbfee3\" (UID: \"bfb66350-fefe-4b07-bd2c-f633a0bbfee3\") "
Nov 03 08:24:43.821048 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:43.821005    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfb66350-fefe-4b07-bd2c-f633a0bbfee3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb66350-fefe-4b07-bd2c-f633a0bbfee3-maas-api-xks-ca\") pod \"bfb66350-fefe-4b07-bd2c-f633a0bbfee3\" (UID: \"bfb66350-fefe-4b07-bd2c-f633a0bbfee3\") "
Nov 03 08:24:43.821164 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:43.821139    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb66350-fefe-4b07-bd2c-f633a0bbfee3-maas-api-openshift-ca" pod "bfb66350-fefe-4b07-bd2c-f633a0bbfee3" (UID: "bfb66350-fefe-4b07-bd2c-f633a0bbfee3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:43.821588 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:43.821556    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfb66350-fefe-4b07-bd2c-f633a0bbfee3-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:43.822983 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:43.822959    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb66350-fefe-4b07-bd2c-f633a0bbfee3-maas-api-xks-ca" pod "bfb66350-fefe-4b07-bd2c-f633a0bbfee3" (UID: "bfb66350-fefe-4b07-bd2c-f633a0bbfee3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:24:43.823067 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:43.823041    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb66350-fefe-4b07-bd2c-f633a0bbfee3-kube-api-access-xt5mz" pod "bfb66350-fefe-4b07-bd2c-f633a0bbfee3" (UID: "bfb66350-fefe-4b07-bd2c-f633a0bbfee3"). InnerVolumeSpecName "kube-api-access-xt5mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:43.922642 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:43.922615    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfb66350-fefe-4b07-bd2c-f633a0bbfee3-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:43.922642 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:43.922644    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt5mz\" (UniqueName: \"kubernetes.io/projected/bfb66350-fefe-4b07-bd2c-f633a0bbfee3-kube-api-access-xt5mz\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.408636 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.408606    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:24:44.444500 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.444464    5070 generic.go:356] "Generic (PLEG): container finished" podID="d915ac09-a22a-4e5d-943a-97cdeb2d9c24" containerID="e07f6abee64d466bfa79ecc820f3501f1b879c5f0b289677d99f154f78040f1a" exitCode=0
Nov 03 08:24:44.444664 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.444569    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb" event={"ID":"d915ac09-a22a-4e5d-943a-97cdeb2d9c24","Type":"ContainerDied","Data":"e07f6abee64d466bfa79ecc820f3501f1b879c5f0b289677d99f154f78040f1a"}
Nov 03 08:24:44.444664 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.444622    5070 scope.go:122] "RemoveContainer" containerID="38969c7bdd9b825821042634a5aac777ff416473f595f646a7ca0043dac7cd4a"
Nov 03 08:24:44.453195 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.453163    5070 generic.go:356] "Generic (PLEG): container finished" podID="a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd" containerID="a9cf4d155950d607fdf623e6e220b88a4bc525f0c17872ffcc38eea4eca176c3" exitCode=0
Nov 03 08:24:44.453486 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.453398    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx" event={"ID":"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd","Type":"ContainerDied","Data":"a9cf4d155950d607fdf623e6e220b88a4bc525f0c17872ffcc38eea4eca176c3"}
Nov 03 08:24:44.458579 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.458550    5070 generic.go:356] "Generic (PLEG): container finished" podID="d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" containerID="e222ed448d5eb626b27ddaf01aa7c0996e4e51c20eed16e3fa24ac4c005a5768" exitCode=0
Nov 03 08:24:44.458680 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.458588    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz" event={"ID":"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb","Type":"ContainerDied","Data":"e222ed448d5eb626b27ddaf01aa7c0996e4e51c20eed16e3fa24ac4c005a5768"}
Nov 03 08:24:44.460606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.460590    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4"
Nov 03 08:24:44.460704 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.460626    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4" event={"ID":"bfb66350-fefe-4b07-bd2c-f633a0bbfee3","Type":"ContainerDied","Data":"e264c83e5d994a8d5b468b895b7071839c0eea09b930b69a1e1e20febc91bba7"}
Nov 03 08:24:44.460704 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.460668    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e264c83e5d994a8d5b468b895b7071839c0eea09b930b69a1e1e20febc91bba7"
Nov 03 08:24:44.462406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.462382    5070 generic.go:356] "Generic (PLEG): container finished" podID="825e9af0-37a8-4d27-9a34-20b2f87e92d5" containerID="52967583fe3b748836a41c5cc2f924344cc62b23ae21445430993d8895d83eb2" exitCode=0
Nov 03 08:24:44.462536 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.462519    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b" event={"ID":"825e9af0-37a8-4d27-9a34-20b2f87e92d5","Type":"ContainerDied","Data":"52967583fe3b748836a41c5cc2f924344cc62b23ae21445430993d8895d83eb2"}
Nov 03 08:24:44.462584 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.462563    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b" event={"ID":"825e9af0-37a8-4d27-9a34-20b2f87e92d5","Type":"ContainerDied","Data":"db366a071ef62b89c30772494f3a3cb0e69b55462adbcdd430e311a607d98b33"}
Nov 03 08:24:44.462655 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.462642    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"
Nov 03 08:24:44.464926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.464902    5070 generic.go:356] "Generic (PLEG): container finished" podID="8fab335e-1024-4086-92ef-49b496f0abc0" containerID="56b5fd507f150bc77dc9b23955016dd8d479ed5d3d9673f1c4d686954d0c2f30" exitCode=0
Nov 03 08:24:44.465024 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.465006    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn" event={"ID":"8fab335e-1024-4086-92ef-49b496f0abc0","Type":"ContainerDied","Data":"56b5fd507f150bc77dc9b23955016dd8d479ed5d3d9673f1c4d686954d0c2f30"}
Nov 03 08:24:44.474644 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.469298    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:24:44.533776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.531426    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-credential-socket" pod "d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" (UID: "d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.533972    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-credential-socket\") pod \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") "
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.534079    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-credential-socket\") pod \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") "
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.534148    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/825e9af0-37a8-4d27-9a34-20b2f87e92d5-kube-api-access-c5zsj\" (UniqueName: \"kubernetes.io/projected/825e9af0-37a8-4d27-9a34-20b2f87e92d5-kube-api-access-c5zsj\") pod \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") "
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.534180    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-data\") pod \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") "
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.534382    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-credential-socket" pod "825e9af0-37a8-4d27-9a34-20b2f87e92d5" (UID: "825e9af0-37a8-4d27-9a34-20b2f87e92d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.534703    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-data" pod "d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" (UID: "d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.535131    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istiod-ca-cert" pod "d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" (UID: "d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.535686    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istiod-ca-cert\") pod \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") "
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.535783    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-workload-certs\") pod \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") "
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.535827    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-envoy\") pod \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") "
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.535885    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-token\" (UniqueName: \"kubernetes.io/projected/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-token\") pod \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") "
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.535935    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-token\" (UniqueName: \"kubernetes.io/projected/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-token\") pod \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") "
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.535988    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-workload-certs\") pod \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") "
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.536046    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istiod-ca-cert\") pod \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") "
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.536076    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-envoy\") pod \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") "
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.536116    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-data\") pod \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") "
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.536143    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-kube-api-access-dzt4m\" (UniqueName: \"kubernetes.io/projected/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-kube-api-access-dzt4m\") pod \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") "
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.536172    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-workload-socket\") pod \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") "
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.536205    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-podinfo\") pod \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") "
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.536265    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-podinfo\") pod \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\" (UID: \"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb\") "
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.536319    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-workload-socket\") pod \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\" (UID: \"825e9af0-37a8-4d27-9a34-20b2f87e92d5\") "
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.537096    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.537121    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.537140    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.537156    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.537662    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-workload-certs" pod "d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" (UID: "d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.538147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.537975    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825e9af0-37a8-4d27-9a34-20b2f87e92d5-kube-api-access-c5zsj" pod "825e9af0-37a8-4d27-9a34-20b2f87e92d5" (UID: "825e9af0-37a8-4d27-9a34-20b2f87e92d5"). InnerVolumeSpecName "kube-api-access-c5zsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:44.539696 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.538276    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istiod-ca-cert" pod "825e9af0-37a8-4d27-9a34-20b2f87e92d5" (UID: "825e9af0-37a8-4d27-9a34-20b2f87e92d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:44.539696 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.538610    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-workload-certs" pod "825e9af0-37a8-4d27-9a34-20b2f87e92d5" (UID: "825e9af0-37a8-4d27-9a34-20b2f87e92d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.540029 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.540002    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-workload-socket" pod "d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" (UID: "d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.540127 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.540014    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-workload-socket" pod "825e9af0-37a8-4d27-9a34-20b2f87e92d5" (UID: "825e9af0-37a8-4d27-9a34-20b2f87e92d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.540243 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.540220    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-kube-api-access-dzt4m" pod "d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" (UID: "d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb"). InnerVolumeSpecName "kube-api-access-dzt4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:44.540435 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.540416    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-data" pod "825e9af0-37a8-4d27-9a34-20b2f87e92d5" (UID: "825e9af0-37a8-4d27-9a34-20b2f87e92d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.546999 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.542745    5070 scope.go:122] "RemoveContainer" containerID="52967583fe3b748836a41c5cc2f924344cc62b23ae21445430993d8895d83eb2"
Nov 03 08:24:44.546999 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.545230    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-token" pod "825e9af0-37a8-4d27-9a34-20b2f87e92d5" (UID: "825e9af0-37a8-4d27-9a34-20b2f87e92d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:44.546999 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.545884    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-token" pod "d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" (UID: "d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:44.546999 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.545888    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-podinfo" pod "d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" (UID: "d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:44.546999 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.546086    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-envoy" pod "825e9af0-37a8-4d27-9a34-20b2f87e92d5" (UID: "825e9af0-37a8-4d27-9a34-20b2f87e92d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.548214 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.547858    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-envoy" pod "d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" (UID: "d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.556320 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.556299    5070 scope.go:122] "RemoveContainer" containerID="52967583fe3b748836a41c5cc2f924344cc62b23ae21445430993d8895d83eb2"
Nov 03 08:24:44.556620 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:44.556599    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52967583fe3b748836a41c5cc2f924344cc62b23ae21445430993d8895d83eb2\": container with ID starting with 52967583fe3b748836a41c5cc2f924344cc62b23ae21445430993d8895d83eb2 not found: ID does not exist" containerID="52967583fe3b748836a41c5cc2f924344cc62b23ae21445430993d8895d83eb2"
Nov 03 08:24:44.556847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.556634    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52967583fe3b748836a41c5cc2f924344cc62b23ae21445430993d8895d83eb2"} err="failed to get container status \"52967583fe3b748836a41c5cc2f924344cc62b23ae21445430993d8895d83eb2\": rpc error: code = NotFound desc = could not find container \"52967583fe3b748836a41c5cc2f924344cc62b23ae21445430993d8895d83eb2\": container with ID starting with 52967583fe3b748836a41c5cc2f924344cc62b23ae21445430993d8895d83eb2 not found: ID does not exist"
Nov 03 08:24:44.561140 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.561116    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-podinfo" pod "825e9af0-37a8-4d27-9a34-20b2f87e92d5" (UID: "825e9af0-37a8-4d27-9a34-20b2f87e92d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:44.573406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.573383    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"]
Nov 03 08:24:44.617149 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.617127    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:24:44.643441 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.642275    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-podinfo\") pod \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") "
Nov 03 08:24:44.643441 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.642806    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-envoy\") pod \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") "
Nov 03 08:24:44.643441 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.642927    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-credential-socket\") pod \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") "
Nov 03 08:24:44.643441 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.643003    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-token\" (UniqueName: \"kubernetes.io/projected/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-token\") pod \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") "
Nov 03 08:24:44.643441 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.643303    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-credential-socket" pod "a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd" (UID: "a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.647376    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-workload-socket" pod "a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd" (UID: "a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.647546    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-workload-socket\") pod \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") "
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.647630    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-workload-certs\") pod \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") "
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.647943    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-workload-certs" pod "a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd" (UID: "a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.648076    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istiod-ca-cert\") pod \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") "
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.648107    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-data\") pod \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") "
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.648683    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istiod-ca-cert" pod "a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd" (UID: "a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.648724    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-data" pod "a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd" (UID: "a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.648842    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-kube-api-access-d2g6t\" (UniqueName: \"kubernetes.io/projected/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-kube-api-access-d2g6t\") pod \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\" (UID: \"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd\") "
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.649429    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-podinfo" pod "a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd" (UID: "a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.650256    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.650278    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.650292    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.650306    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.650320    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.650333    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.650348    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.650361    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.650374    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.650386    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.650401    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzt4m\" (UniqueName: \"kubernetes.io/projected/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-kube-api-access-dzt4m\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.650414    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.650427    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/825e9af0-37a8-4d27-9a34-20b2f87e92d5-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.650439    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.650452    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/825e9af0-37a8-4d27-9a34-20b2f87e92d5-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.650464    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.650477    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.650491    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.650504    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.650517    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5zsj\" (UniqueName: \"kubernetes.io/projected/825e9af0-37a8-4d27-9a34-20b2f87e92d5-kube-api-access-c5zsj\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.653321    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-kube-api-access-d2g6t" pod "a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd" (UID: "a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd"). InnerVolumeSpecName "kube-api-access-d2g6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.654188    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-envoy" pod "a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd" (UID: "a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.656623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.655952    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-token" pod "a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd" (UID: "a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:44.693492 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.693470    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:24:44.703301 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.701921    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:24:44.751496 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.751462    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-podinfo\") pod \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") "
Nov 03 08:24:44.751724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.751528    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istiod-ca-cert\") pod \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") "
Nov 03 08:24:44.751724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.751575    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-envoy\") pod \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") "
Nov 03 08:24:44.751724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.751604    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fab335e-1024-4086-92ef-49b496f0abc0-kube-api-access-blqml\" (UniqueName: \"kubernetes.io/projected/8fab335e-1024-4086-92ef-49b496f0abc0-kube-api-access-blqml\") pod \"8fab335e-1024-4086-92ef-49b496f0abc0\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") "
Nov 03 08:24:44.751724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.751637    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-workload-socket\") pod \"8fab335e-1024-4086-92ef-49b496f0abc0\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") "
Nov 03 08:24:44.751724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.751676    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-credential-socket\") pod \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") "
Nov 03 08:24:44.751724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.751712    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-workload-socket\") pod \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") "
Nov 03 08:24:44.752087 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.751781    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fab335e-1024-4086-92ef-49b496f0abc0-istio-token\" (UniqueName: \"kubernetes.io/projected/8fab335e-1024-4086-92ef-49b496f0abc0-istio-token\") pod \"8fab335e-1024-4086-92ef-49b496f0abc0\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") "
Nov 03 08:24:44.752087 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.751830    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-credential-socket\") pod \"8fab335e-1024-4086-92ef-49b496f0abc0\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") "
Nov 03 08:24:44.752087 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.751991    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-istio-envoy\") pod \"8fab335e-1024-4086-92ef-49b496f0abc0\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") "
Nov 03 08:24:44.752087 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.752022    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-istio-data\") pod \"8fab335e-1024-4086-92ef-49b496f0abc0\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") "
Nov 03 08:24:44.752087 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.752067    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-data\") pod \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") "
Nov 03 08:24:44.752448 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.752136    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-workload-certs\") pod \"8fab335e-1024-4086-92ef-49b496f0abc0\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") "
Nov 03 08:24:44.752448 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.752187    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-kube-api-access-9hg2s\" (UniqueName: \"kubernetes.io/projected/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-kube-api-access-9hg2s\") pod \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") "
Nov 03 08:24:44.752448 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.752218    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-workload-certs\") pod \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") "
Nov 03 08:24:44.752448 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.752279    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-token\" (UniqueName: \"kubernetes.io/projected/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-token\") pod \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\" (UID: \"d915ac09-a22a-4e5d-943a-97cdeb2d9c24\") "
Nov 03 08:24:44.753874 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.753844    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fab335e-1024-4086-92ef-49b496f0abc0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fab335e-1024-4086-92ef-49b496f0abc0-istio-podinfo\") pod \"8fab335e-1024-4086-92ef-49b496f0abc0\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") "
Nov 03 08:24:44.754032 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.753915    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fab335e-1024-4086-92ef-49b496f0abc0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fab335e-1024-4086-92ef-49b496f0abc0-istiod-ca-cert\") pod \"8fab335e-1024-4086-92ef-49b496f0abc0\" (UID: \"8fab335e-1024-4086-92ef-49b496f0abc0\") "
Nov 03 08:24:44.762812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.754500    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2g6t\" (UniqueName: \"kubernetes.io/projected/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-kube-api-access-d2g6t\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.762812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.754529    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.762812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.754543    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.762812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.756287    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istiod-ca-cert" pod "d915ac09-a22a-4e5d-943a-97cdeb2d9c24" (UID: "d915ac09-a22a-4e5d-943a-97cdeb2d9c24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:44.762812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.757174    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-credential-socket" pod "d915ac09-a22a-4e5d-943a-97cdeb2d9c24" (UID: "d915ac09-a22a-4e5d-943a-97cdeb2d9c24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.762812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.759067    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-envoy" pod "d915ac09-a22a-4e5d-943a-97cdeb2d9c24" (UID: "d915ac09-a22a-4e5d-943a-97cdeb2d9c24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.762812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.759520    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-workload-socket" pod "d915ac09-a22a-4e5d-943a-97cdeb2d9c24" (UID: "d915ac09-a22a-4e5d-943a-97cdeb2d9c24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.762812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.759703    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fab335e-1024-4086-92ef-49b496f0abc0-kube-api-access-blqml" pod "8fab335e-1024-4086-92ef-49b496f0abc0" (UID: "8fab335e-1024-4086-92ef-49b496f0abc0"). InnerVolumeSpecName "kube-api-access-blqml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:44.762812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.759728    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-credential-socket" pod "8fab335e-1024-4086-92ef-49b496f0abc0" (UID: "8fab335e-1024-4086-92ef-49b496f0abc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.762812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.760012    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-workload-certs" pod "d915ac09-a22a-4e5d-943a-97cdeb2d9c24" (UID: "d915ac09-a22a-4e5d-943a-97cdeb2d9c24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.762812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.760214    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-data" pod "d915ac09-a22a-4e5d-943a-97cdeb2d9c24" (UID: "d915ac09-a22a-4e5d-943a-97cdeb2d9c24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.762812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.760390    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-istio-data" pod "8fab335e-1024-4086-92ef-49b496f0abc0" (UID: "8fab335e-1024-4086-92ef-49b496f0abc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.762812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.760908    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-workload-socket" pod "8fab335e-1024-4086-92ef-49b496f0abc0" (UID: "8fab335e-1024-4086-92ef-49b496f0abc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.762812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.761086    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-workload-certs" pod "8fab335e-1024-4086-92ef-49b496f0abc0" (UID: "8fab335e-1024-4086-92ef-49b496f0abc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.762812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.762355    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fab335e-1024-4086-92ef-49b496f0abc0-istio-podinfo" pod "8fab335e-1024-4086-92ef-49b496f0abc0" (UID: "8fab335e-1024-4086-92ef-49b496f0abc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:44.763639 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.763611    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fab335e-1024-4086-92ef-49b496f0abc0-istio-token" pod "8fab335e-1024-4086-92ef-49b496f0abc0" (UID: "8fab335e-1024-4086-92ef-49b496f0abc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:44.767185 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.767163    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-kube-api-access-9hg2s" pod "d915ac09-a22a-4e5d-943a-97cdeb2d9c24" (UID: "d915ac09-a22a-4e5d-943a-97cdeb2d9c24"). InnerVolumeSpecName "kube-api-access-9hg2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:44.767185 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.767160    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-istio-envoy" pod "8fab335e-1024-4086-92ef-49b496f0abc0" (UID: "8fab335e-1024-4086-92ef-49b496f0abc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:24:44.767896 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.767227    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-podinfo" pod "d915ac09-a22a-4e5d-943a-97cdeb2d9c24" (UID: "d915ac09-a22a-4e5d-943a-97cdeb2d9c24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:24:44.768402 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.768382    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-token" pod "d915ac09-a22a-4e5d-943a-97cdeb2d9c24" (UID: "d915ac09-a22a-4e5d-943a-97cdeb2d9c24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:44.770866 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.770844    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fab335e-1024-4086-92ef-49b496f0abc0-istiod-ca-cert" pod "8fab335e-1024-4086-92ef-49b496f0abc0" (UID: "8fab335e-1024-4086-92ef-49b496f0abc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:44.815153 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.815127    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"]
Nov 03 08:24:44.819114 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.819092    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ftr8b"]
Nov 03 08:24:44.855489 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.855470    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fab335e-1024-4086-92ef-49b496f0abc0-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.855568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.855491    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.855568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.855500    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.855568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.855509    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.855568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.855519    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blqml\" (UniqueName: \"kubernetes.io/projected/8fab335e-1024-4086-92ef-49b496f0abc0-kube-api-access-blqml\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.855568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.855528    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.855568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.855536    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.855568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.855546    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.855568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.855554    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fab335e-1024-4086-92ef-49b496f0abc0-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.855568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.855562    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.855899 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.855572    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.855899 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.855580    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.855899 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.855589    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.855899 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.855598    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fab335e-1024-4086-92ef-49b496f0abc0-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.855899 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.855606    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hg2s\" (UniqueName: \"kubernetes.io/projected/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-kube-api-access-9hg2s\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.855899 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.855614    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.855899 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.855623    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d915ac09-a22a-4e5d-943a-97cdeb2d9c24-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:44.855899 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:44.855631    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fab335e-1024-4086-92ef-49b496f0abc0-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:45.486324 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.486289    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"
Nov 03 08:24:45.486324 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.486300    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn" event={"ID":"8fab335e-1024-4086-92ef-49b496f0abc0","Type":"ContainerDied","Data":"93ce3e6108e1b4cae36a596a588460cd5cc1eaf290e8afb729a795970aa90062"}
Nov 03 08:24:45.486584 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.486397    5070 scope.go:122] "RemoveContainer" containerID="56b5fd507f150bc77dc9b23955016dd8d479ed5d3d9673f1c4d686954d0c2f30"
Nov 03 08:24:45.488055 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.488032    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"
Nov 03 08:24:45.488167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.488034    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb" event={"ID":"d915ac09-a22a-4e5d-943a-97cdeb2d9c24","Type":"ContainerDied","Data":"9400476ef3dd3ad4541ed7f2b386b3de41dd986bd900039a53bb9da225960a93"}
Nov 03 08:24:45.490672 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.490651    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"
Nov 03 08:24:45.490813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.490687    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx" event={"ID":"a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd","Type":"ContainerDied","Data":"4c79800a66cf207083df2a35af46cc33f9ff50b7d6d2d13272023079fd5999a9"}
Nov 03 08:24:45.498645 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.498623    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"
Nov 03 08:24:45.498740 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.498660    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz" event={"ID":"d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb","Type":"ContainerDied","Data":"3c34707ae33c4775ff509e7a7059aa91f0cd1eb3e130b7f863836f1e86a05ca8"}
Nov 03 08:24:45.501682 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.501665    5070 scope.go:122] "RemoveContainer" containerID="e07f6abee64d466bfa79ecc820f3501f1b879c5f0b289677d99f154f78040f1a"
Nov 03 08:24:45.513621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.513598    5070 scope.go:122] "RemoveContainer" containerID="a9cf4d155950d607fdf623e6e220b88a4bc525f0c17872ffcc38eea4eca176c3"
Nov 03 08:24:45.519673 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.519640    5070 patch_prober.go:29] interesting pod/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:24:45.519836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.519685    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb" podUID="d915ac09-a22a-4e5d-943a-97cdeb2d9c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:24:45.524341 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.524317    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"]
Nov 03 08:24:45.525263 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.525244    5070 scope.go:122] "RemoveContainer" containerID="e222ed448d5eb626b27ddaf01aa7c0996e4e51c20eed16e3fa24ac4c005a5768"
Nov 03 08:24:45.528634 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.528615    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-ctkmn"]
Nov 03 08:24:45.538713 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.538693    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"]
Nov 03 08:24:45.545219 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.545201    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-gdfpb"]
Nov 03 08:24:45.554213 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.554182    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"]
Nov 03 08:24:45.560140 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.560120    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-trjvx"]
Nov 03 08:24:45.572800 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.568657    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"]
Nov 03 08:24:45.586805 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:45.581716    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b8a152-openshift-default-6db4f7f78c-mffqz"]
Nov 03 08:24:46.026661 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:46.026628    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="825e9af0-37a8-4d27-9a34-20b2f87e92d5" path="/var/lib/kubelet/pods/825e9af0-37a8-4d27-9a34-20b2f87e92d5/volumes"
Nov 03 08:24:46.027300 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:46.027277    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fab335e-1024-4086-92ef-49b496f0abc0" path="/var/lib/kubelet/pods/8fab335e-1024-4086-92ef-49b496f0abc0/volumes"
Nov 03 08:24:46.027804 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:46.027785    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd" path="/var/lib/kubelet/pods/a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd/volumes"
Nov 03 08:24:46.028587 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:46.028565    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" path="/var/lib/kubelet/pods/d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb/volumes"
Nov 03 08:24:46.029100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:46.029079    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d915ac09-a22a-4e5d-943a-97cdeb2d9c24" path="/var/lib/kubelet/pods/d915ac09-a22a-4e5d-943a-97cdeb2d9c24/volumes"
Nov 03 08:24:48.187074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:48.187030    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd" podUID="cb961482-e401-47f5-8aff-f604b364d631" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:24:49.864413 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:49.864378    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2"]
Nov 03 08:24:49.869901 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:49.869879    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-3e88b7-nh8g2"]
Nov 03 08:24:50.026516 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:50.026486    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2484ae5d-667e-4ee2-a1fa-01c61ded3d45" path="/var/lib/kubelet/pods/2484ae5d-667e-4ee2-a1fa-01c61ded3d45/volumes"
Nov 03 08:24:52.051288 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:52.051256    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:24:55.192843 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.192806    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm"]
Nov 03 08:24:55.193615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193598    5070 cpu_manager.go:414] "containerMap: removing container" podUID="d915ac09-a22a-4e5d-943a-97cdeb2d9c24" containerName="istio-proxy"
Nov 03 08:24:55.193667 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193617    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d915ac09-a22a-4e5d-943a-97cdeb2d9c24" containerName="istio-proxy"
Nov 03 08:24:55.193667 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193632    5070 cpu_manager.go:414] "containerMap: removing container" podUID="8fab335e-1024-4086-92ef-49b496f0abc0" containerName="istio-proxy"
Nov 03 08:24:55.193667 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193637    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fab335e-1024-4086-92ef-49b496f0abc0" containerName="istio-proxy"
Nov 03 08:24:55.193667 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193652    5070 cpu_manager.go:414] "containerMap: removing container" podUID="d915ac09-a22a-4e5d-943a-97cdeb2d9c24" containerName="istio-proxy"
Nov 03 08:24:55.193667 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193658    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d915ac09-a22a-4e5d-943a-97cdeb2d9c24" containerName="istio-proxy"
Nov 03 08:24:55.193667 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193668    5070 cpu_manager.go:414] "containerMap: removing container" podUID="0c070983-a6e2-41ce-aff5-8526f7d99375" containerName="revoke-keys"
Nov 03 08:24:55.193892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193674    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c070983-a6e2-41ce-aff5-8526f7d99375" containerName="revoke-keys"
Nov 03 08:24:55.193892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193695    5070 cpu_manager.go:414] "containerMap: removing container" podUID="d0820001-c7d2-49f8-afa6-79038e19c795" containerName="revoke-keys"
Nov 03 08:24:55.193892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193700    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0820001-c7d2-49f8-afa6-79038e19c795" containerName="revoke-keys"
Nov 03 08:24:55.193892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193715    5070 cpu_manager.go:414] "containerMap: removing container" podUID="825e9af0-37a8-4d27-9a34-20b2f87e92d5" containerName="istio-proxy"
Nov 03 08:24:55.193892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193719    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="825e9af0-37a8-4d27-9a34-20b2f87e92d5" containerName="istio-proxy"
Nov 03 08:24:55.193892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193735    5070 cpu_manager.go:414] "containerMap: removing container" podUID="bfb66350-fefe-4b07-bd2c-f633a0bbfee3" containerName="revoke-keys"
Nov 03 08:24:55.193892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193740    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfb66350-fefe-4b07-bd2c-f633a0bbfee3" containerName="revoke-keys"
Nov 03 08:24:55.193892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193748    5070 cpu_manager.go:414] "containerMap: removing container" podUID="d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" containerName="istio-proxy"
Nov 03 08:24:55.193892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193753    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" containerName="istio-proxy"
Nov 03 08:24:55.193892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193774    5070 cpu_manager.go:414] "containerMap: removing container" podUID="a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd" containerName="istio-proxy"
Nov 03 08:24:55.193892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193779    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd" containerName="istio-proxy"
Nov 03 08:24:55.193892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193866    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfb66350-fefe-4b07-bd2c-f633a0bbfee3" containerName="revoke-keys"
Nov 03 08:24:55.193892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193877    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="d75f2a3f-2d62-4f7f-a449-eba6bb6cfceb" containerName="istio-proxy"
Nov 03 08:24:55.193892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193883    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c9e812-cff9-46b7-8ee1-1bea23dcd7fd" containerName="istio-proxy"
Nov 03 08:24:55.193892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193890    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="d915ac09-a22a-4e5d-943a-97cdeb2d9c24" containerName="istio-proxy"
Nov 03 08:24:55.193892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193896    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fab335e-1024-4086-92ef-49b496f0abc0" containerName="istio-proxy"
Nov 03 08:24:55.194382 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193903    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="d915ac09-a22a-4e5d-943a-97cdeb2d9c24" containerName="istio-proxy"
Nov 03 08:24:55.194382 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193910    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c070983-a6e2-41ce-aff5-8526f7d99375" containerName="revoke-keys"
Nov 03 08:24:55.194382 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193919    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0820001-c7d2-49f8-afa6-79038e19c795" containerName="revoke-keys"
Nov 03 08:24:55.194382 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.193926    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="825e9af0-37a8-4d27-9a34-20b2f87e92d5" containerName="istio-proxy"
Nov 03 08:24:55.199267 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.199246    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp"]
Nov 03 08:24:55.199425 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.199371    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm"
Nov 03 08:24:55.202666 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.202254    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:24:55.202666 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.202262    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:24:55.202666 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.202325    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:24:55.202666 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.202337    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:24:55.204136 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.204117    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp"
Nov 03 08:24:55.204861 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.204841    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm"]
Nov 03 08:24:55.223541 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.223510    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp"]
Nov 03 08:24:55.294291 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.294259    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd9tq\" (UniqueName: \"kubernetes.io/projected/42228923-0917-4753-af54-c3fc849be091-kube-api-access-zd9tq\") pod \"maas-api-revoke-sub-9162aaae-f52hm\" (UID: \"42228923-0917-4753-af54-c3fc849be091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm"
Nov 03 08:24:55.294420 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.294368    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tbjt\" (UniqueName: \"kubernetes.io/projected/506fb467-77cf-4b49-8e99-b787b98b490f-kube-api-access-5tbjt\") pod \"maas-api-revoke-sub-157f42c1-mc5rp\" (UID: \"506fb467-77cf-4b49-8e99-b787b98b490f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp"
Nov 03 08:24:55.294420 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.294393    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42228923-0917-4753-af54-c3fc849be091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9162aaae-f52hm\" (UID: \"42228923-0917-4753-af54-c3fc849be091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm"
Nov 03 08:24:55.294546 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.294533    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506fb467-77cf-4b49-8e99-b787b98b490f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-157f42c1-mc5rp\" (UID: \"506fb467-77cf-4b49-8e99-b787b98b490f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp"
Nov 03 08:24:55.294607 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.294595    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42228923-0917-4753-af54-c3fc849be091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9162aaae-f52hm\" (UID: \"42228923-0917-4753-af54-c3fc849be091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm"
Nov 03 08:24:55.294657 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.294625    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506fb467-77cf-4b49-8e99-b787b98b490f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-157f42c1-mc5rp\" (UID: \"506fb467-77cf-4b49-8e99-b787b98b490f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp"
Nov 03 08:24:55.395796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.395738    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42228923-0917-4753-af54-c3fc849be091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9162aaae-f52hm\" (UID: \"42228923-0917-4753-af54-c3fc849be091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm"
Nov 03 08:24:55.395956 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.395819    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506fb467-77cf-4b49-8e99-b787b98b490f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-157f42c1-mc5rp\" (UID: \"506fb467-77cf-4b49-8e99-b787b98b490f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp"
Nov 03 08:24:55.395956 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.395899    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd9tq\" (UniqueName: \"kubernetes.io/projected/42228923-0917-4753-af54-c3fc849be091-kube-api-access-zd9tq\") pod \"maas-api-revoke-sub-9162aaae-f52hm\" (UID: \"42228923-0917-4753-af54-c3fc849be091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm"
Nov 03 08:24:55.396082 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.395960    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tbjt\" (UniqueName: \"kubernetes.io/projected/506fb467-77cf-4b49-8e99-b787b98b490f-kube-api-access-5tbjt\") pod \"maas-api-revoke-sub-157f42c1-mc5rp\" (UID: \"506fb467-77cf-4b49-8e99-b787b98b490f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp"
Nov 03 08:24:55.396082 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.395985    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42228923-0917-4753-af54-c3fc849be091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9162aaae-f52hm\" (UID: \"42228923-0917-4753-af54-c3fc849be091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm"
Nov 03 08:24:55.396207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.396093    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506fb467-77cf-4b49-8e99-b787b98b490f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-157f42c1-mc5rp\" (UID: \"506fb467-77cf-4b49-8e99-b787b98b490f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp"
Nov 03 08:24:55.396509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.396485    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42228923-0917-4753-af54-c3fc849be091-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9162aaae-f52hm\" (UID: \"42228923-0917-4753-af54-c3fc849be091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm"
Nov 03 08:24:55.396600 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.396561    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506fb467-77cf-4b49-8e99-b787b98b490f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-157f42c1-mc5rp\" (UID: \"506fb467-77cf-4b49-8e99-b787b98b490f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp"
Nov 03 08:24:55.398384 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.398356    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42228923-0917-4753-af54-c3fc849be091-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9162aaae-f52hm\" (UID: \"42228923-0917-4753-af54-c3fc849be091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm"
Nov 03 08:24:55.398532 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.398512    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506fb467-77cf-4b49-8e99-b787b98b490f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-157f42c1-mc5rp\" (UID: \"506fb467-77cf-4b49-8e99-b787b98b490f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp"
Nov 03 08:24:55.402744 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.402721    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tbjt\" (UniqueName: \"kubernetes.io/projected/506fb467-77cf-4b49-8e99-b787b98b490f-kube-api-access-5tbjt\") pod \"maas-api-revoke-sub-157f42c1-mc5rp\" (UID: \"506fb467-77cf-4b49-8e99-b787b98b490f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp"
Nov 03 08:24:55.402903 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.402883    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd9tq\" (UniqueName: \"kubernetes.io/projected/42228923-0917-4753-af54-c3fc849be091-kube-api-access-zd9tq\") pod \"maas-api-revoke-sub-9162aaae-f52hm\" (UID: \"42228923-0917-4753-af54-c3fc849be091\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm"
Nov 03 08:24:55.519014 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.518965    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm"
Nov 03 08:24:55.533141 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.533119    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp"
Nov 03 08:24:55.842093 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.842062    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm"]
Nov 03 08:24:55.842456 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:24:55.842429    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42228923_0917_4753_af54_c3fc849be091.slice/crio-b99656c81069f2da0e42704195d629cbc25f65631b963ee24acfcac3cf13a3fe WatchSource:0}: Error finding container b99656c81069f2da0e42704195d629cbc25f65631b963ee24acfcac3cf13a3fe: Status 404 returned error can't find the container with id b99656c81069f2da0e42704195d629cbc25f65631b963ee24acfcac3cf13a3fe
Nov 03 08:24:55.852414 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:55.852394    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp"]
Nov 03 08:24:55.856040 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:24:55.856017    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506fb467_77cf_4b49_8e99_b787b98b490f.slice/crio-e3433725cd059c41a5c1a2ed5cd73d4015c4979e6f12be66168f8f252be735f5 WatchSource:0}: Error finding container e3433725cd059c41a5c1a2ed5cd73d4015c4979e6f12be66168f8f252be735f5: Status 404 returned error can't find the container with id e3433725cd059c41a5c1a2ed5cd73d4015c4979e6f12be66168f8f252be735f5
Nov 03 08:24:56.627703 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:56.627656    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"]
Nov 03 08:24:56.628190 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:56.628049    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89" podUID="bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" containerName="istio-proxy" containerID="cri-o://b1401b6134a3ead2bdae6e4103b8040456f68f7acc759b0a14112e64ab659688" gracePeriod=30
Nov 03 08:24:56.697583 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:56.697542    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm" event={"ID":"42228923-0917-4753-af54-c3fc849be091","Type":"ContainerStarted","Data":"2d548fe6af73f0948b4b2e1491c2aa364fe93cf53a5ed11efbd293807aad58bc"}
Nov 03 08:24:56.697732 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:56.697606    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm" event={"ID":"42228923-0917-4753-af54-c3fc849be091","Type":"ContainerStarted","Data":"b99656c81069f2da0e42704195d629cbc25f65631b963ee24acfcac3cf13a3fe"}
Nov 03 08:24:56.699954 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:56.699924    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp" event={"ID":"506fb467-77cf-4b49-8e99-b787b98b490f","Type":"ContainerStarted","Data":"7e5b490de34ee80bd519cb3455730a44cf45e883f30f480cc6bf0e599e7ba9e6"}
Nov 03 08:24:56.700160 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:56.700143    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp" event={"ID":"506fb467-77cf-4b49-8e99-b787b98b490f","Type":"ContainerStarted","Data":"e3433725cd059c41a5c1a2ed5cd73d4015c4979e6f12be66168f8f252be735f5"}
Nov 03 08:24:56.712032 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:56.711967    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm" podStartSLOduration=1.711953082 podStartE2EDuration="1.711953082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:24:56.710971723 +0000 UTC m=+1894.825435654" watchObservedRunningTime="2026-11-03 08:24:56.711953082 +0000 UTC m=+1894.826417016"
Nov 03 08:24:56.782781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:56.781615    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp" podStartSLOduration=1.781596492 podStartE2EDuration="1.781596492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:24:56.726164894 +0000 UTC m=+1894.840628829" watchObservedRunningTime="2026-11-03 08:24:56.781596492 +0000 UTC m=+1894.896060426"
Nov 03 08:24:57.710060 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:57.710024    5070 generic.go:356] "Generic (PLEG): container finished" podID="42228923-0917-4753-af54-c3fc849be091" containerID="2d548fe6af73f0948b4b2e1491c2aa364fe93cf53a5ed11efbd293807aad58bc" exitCode=0
Nov 03 08:24:57.710453 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:57.710121    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm" event={"ID":"42228923-0917-4753-af54-c3fc849be091","Type":"ContainerDied","Data":"2d548fe6af73f0948b4b2e1491c2aa364fe93cf53a5ed11efbd293807aad58bc"}
Nov 03 08:24:57.711818 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:57.711789    5070 generic.go:356] "Generic (PLEG): container finished" podID="506fb467-77cf-4b49-8e99-b787b98b490f" containerID="7e5b490de34ee80bd519cb3455730a44cf45e883f30f480cc6bf0e599e7ba9e6" exitCode=0
Nov 03 08:24:57.711952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:57.711874    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp" event={"ID":"506fb467-77cf-4b49-8e99-b787b98b490f","Type":"ContainerDied","Data":"7e5b490de34ee80bd519cb3455730a44cf45e883f30f480cc6bf0e599e7ba9e6"}
Nov 03 08:24:58.187788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.187727    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd" podUID="cb961482-e401-47f5-8aff-f604b364d631" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:24:58.506034 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.506007    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd"
Nov 03 08:24:58.600915 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.600874    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb961482-e401-47f5-8aff-f604b364d631-plugins-config-volume" pod "cb961482-e401-47f5-8aff-f604b364d631" (UID: "cb961482-e401-47f5-8aff-f604b364d631"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:58.601224 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.601198    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb961482-e401-47f5-8aff-f604b364d631-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb961482-e401-47f5-8aff-f604b364d631-plugins-config-volume\") pod \"cb961482-e401-47f5-8aff-f604b364d631\" (UID: \"cb961482-e401-47f5-8aff-f604b364d631\") "
Nov 03 08:24:58.601547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.601527    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb961482-e401-47f5-8aff-f604b364d631-kube-api-access-cpwmr\" (UniqueName: \"kubernetes.io/projected/cb961482-e401-47f5-8aff-f604b364d631-kube-api-access-cpwmr\") pod \"cb961482-e401-47f5-8aff-f604b364d631\" (UID: \"cb961482-e401-47f5-8aff-f604b364d631\") "
Nov 03 08:24:58.606394 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.606364    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb961482-e401-47f5-8aff-f604b364d631-kube-api-access-cpwmr" pod "cb961482-e401-47f5-8aff-f604b364d631" (UID: "cb961482-e401-47f5-8aff-f604b364d631"). InnerVolumeSpecName "kube-api-access-cpwmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:58.704488 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.704451    5070 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb961482-e401-47f5-8aff-f604b364d631-plugins-config-volume\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:58.704488 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.704480    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpwmr\" (UniqueName: \"kubernetes.io/projected/cb961482-e401-47f5-8aff-f604b364d631-kube-api-access-cpwmr\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:58.721908 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.721841    5070 generic.go:356] "Generic (PLEG): container finished" podID="cb961482-e401-47f5-8aff-f604b364d631" containerID="493c820308728c3e98ad907864a52d22363893328c3b730567fe658c759f2a07" exitCode=0
Nov 03 08:24:58.721908 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.721892    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd"
Nov 03 08:24:58.722345 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.721943    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd" event={"ID":"cb961482-e401-47f5-8aff-f604b364d631","Type":"ContainerDied","Data":"493c820308728c3e98ad907864a52d22363893328c3b730567fe658c759f2a07"}
Nov 03 08:24:58.722345 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.722006    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd" event={"ID":"cb961482-e401-47f5-8aff-f604b364d631","Type":"ContainerDied","Data":"612729ff76eca47f2a71e185ed5b5739c2e9e17e59b380030478d57b2852522e"}
Nov 03 08:24:58.722345 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.722035    5070 scope.go:122] "RemoveContainer" containerID="493c820308728c3e98ad907864a52d22363893328c3b730567fe658c759f2a07"
Nov 03 08:24:58.736219 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.736199    5070 scope.go:122] "RemoveContainer" containerID="493c820308728c3e98ad907864a52d22363893328c3b730567fe658c759f2a07"
Nov 03 08:24:58.736525 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:24:58.736505    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"493c820308728c3e98ad907864a52d22363893328c3b730567fe658c759f2a07\": container with ID starting with 493c820308728c3e98ad907864a52d22363893328c3b730567fe658c759f2a07 not found: ID does not exist" containerID="493c820308728c3e98ad907864a52d22363893328c3b730567fe658c759f2a07"
Nov 03 08:24:58.736630 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.736532    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"493c820308728c3e98ad907864a52d22363893328c3b730567fe658c759f2a07"} err="failed to get container status \"493c820308728c3e98ad907864a52d22363893328c3b730567fe658c759f2a07\": rpc error: code = NotFound desc = could not find container \"493c820308728c3e98ad907864a52d22363893328c3b730567fe658c759f2a07\": container with ID starting with 493c820308728c3e98ad907864a52d22363893328c3b730567fe658c759f2a07 not found: ID does not exist"
Nov 03 08:24:58.745047 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.745024    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd"]
Nov 03 08:24:58.754235 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.754211    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-3e88b7-5fb97cd65f-fdlnd"]
Nov 03 08:24:58.877432 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.877404    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp"
Nov 03 08:24:58.902520 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.902493    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm"
Nov 03 08:24:58.906290 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.906262    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/506fb467-77cf-4b49-8e99-b787b98b490f-kube-api-access-5tbjt\" (UniqueName: \"kubernetes.io/projected/506fb467-77cf-4b49-8e99-b787b98b490f-kube-api-access-5tbjt\") pod \"506fb467-77cf-4b49-8e99-b787b98b490f\" (UID: \"506fb467-77cf-4b49-8e99-b787b98b490f\") "
Nov 03 08:24:58.906451 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.906302    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/506fb467-77cf-4b49-8e99-b787b98b490f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506fb467-77cf-4b49-8e99-b787b98b490f-maas-api-openshift-ca\") pod \"506fb467-77cf-4b49-8e99-b787b98b490f\" (UID: \"506fb467-77cf-4b49-8e99-b787b98b490f\") "
Nov 03 08:24:58.906451 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.906374    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/506fb467-77cf-4b49-8e99-b787b98b490f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506fb467-77cf-4b49-8e99-b787b98b490f-maas-api-xks-ca\") pod \"506fb467-77cf-4b49-8e99-b787b98b490f\" (UID: \"506fb467-77cf-4b49-8e99-b787b98b490f\") "
Nov 03 08:24:58.906685 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.906663    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506fb467-77cf-4b49-8e99-b787b98b490f-maas-api-openshift-ca" pod "506fb467-77cf-4b49-8e99-b787b98b490f" (UID: "506fb467-77cf-4b49-8e99-b787b98b490f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:58.906972 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.906949    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506fb467-77cf-4b49-8e99-b787b98b490f-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:58.908593 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.908568    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/506fb467-77cf-4b49-8e99-b787b98b490f-maas-api-xks-ca" pod "506fb467-77cf-4b49-8e99-b787b98b490f" (UID: "506fb467-77cf-4b49-8e99-b787b98b490f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:24:58.909087 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:58.909057    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506fb467-77cf-4b49-8e99-b787b98b490f-kube-api-access-5tbjt" pod "506fb467-77cf-4b49-8e99-b787b98b490f" (UID: "506fb467-77cf-4b49-8e99-b787b98b490f"). InnerVolumeSpecName "kube-api-access-5tbjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:59.007614 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.007543    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42228923-0917-4753-af54-c3fc849be091-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42228923-0917-4753-af54-c3fc849be091-maas-api-xks-ca\") pod \"42228923-0917-4753-af54-c3fc849be091\" (UID: \"42228923-0917-4753-af54-c3fc849be091\") "
Nov 03 08:24:59.007750 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.007683    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42228923-0917-4753-af54-c3fc849be091-kube-api-access-zd9tq\" (UniqueName: \"kubernetes.io/projected/42228923-0917-4753-af54-c3fc849be091-kube-api-access-zd9tq\") pod \"42228923-0917-4753-af54-c3fc849be091\" (UID: \"42228923-0917-4753-af54-c3fc849be091\") "
Nov 03 08:24:59.007847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.007798    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42228923-0917-4753-af54-c3fc849be091-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42228923-0917-4753-af54-c3fc849be091-maas-api-openshift-ca\") pod \"42228923-0917-4753-af54-c3fc849be091\" (UID: \"42228923-0917-4753-af54-c3fc849be091\") "
Nov 03 08:24:59.008090 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.008067    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42228923-0917-4753-af54-c3fc849be091-maas-api-openshift-ca" pod "42228923-0917-4753-af54-c3fc849be091" (UID: "42228923-0917-4753-af54-c3fc849be091"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:24:59.008166 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.008119    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tbjt\" (UniqueName: \"kubernetes.io/projected/506fb467-77cf-4b49-8e99-b787b98b490f-kube-api-access-5tbjt\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:59.008166 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.008134    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506fb467-77cf-4b49-8e99-b787b98b490f-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:59.009664 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.009642    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42228923-0917-4753-af54-c3fc849be091-kube-api-access-zd9tq" pod "42228923-0917-4753-af54-c3fc849be091" (UID: "42228923-0917-4753-af54-c3fc849be091"). InnerVolumeSpecName "kube-api-access-zd9tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:24:59.009721 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.009645    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42228923-0917-4753-af54-c3fc849be091-maas-api-xks-ca" pod "42228923-0917-4753-af54-c3fc849be091" (UID: "42228923-0917-4753-af54-c3fc849be091"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:24:59.110129 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.110103    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd9tq\" (UniqueName: \"kubernetes.io/projected/42228923-0917-4753-af54-c3fc849be091-kube-api-access-zd9tq\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:59.110250 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.110150    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42228923-0917-4753-af54-c3fc849be091-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:59.110250 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.110169    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42228923-0917-4753-af54-c3fc849be091-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:24:59.611609 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.611573    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" containerID="cri-o://414accb291de910b47a17b89016eed4be600c142606fff6de269ea0f69757017" gracePeriod=45
Nov 03 08:24:59.716820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.716780    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": dial tcp 10.132.0.102:8000: connect: connection refused"
Nov 03 08:24:59.733200 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.733177    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp"
Nov 03 08:24:59.733599 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.733349    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp" event={"ID":"506fb467-77cf-4b49-8e99-b787b98b490f","Type":"ContainerDied","Data":"e3433725cd059c41a5c1a2ed5cd73d4015c4979e6f12be66168f8f252be735f5"}
Nov 03 08:24:59.733599 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.733392    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3433725cd059c41a5c1a2ed5cd73d4015c4979e6f12be66168f8f252be735f5"
Nov 03 08:24:59.735098 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.735078    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm"
Nov 03 08:24:59.735218 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.735108    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm" event={"ID":"42228923-0917-4753-af54-c3fc849be091","Type":"ContainerDied","Data":"b99656c81069f2da0e42704195d629cbc25f65631b963ee24acfcac3cf13a3fe"}
Nov 03 08:24:59.735218 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.735149    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b99656c81069f2da0e42704195d629cbc25f65631b963ee24acfcac3cf13a3fe"
Nov 03 08:24:59.736898 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.736873    5070 generic.go:356] "Generic (PLEG): container finished" podID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerID="414accb291de910b47a17b89016eed4be600c142606fff6de269ea0f69757017" exitCode=0
Nov 03 08:24:59.737015 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.736942    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" event={"ID":"d172e36b-89d5-43d3-9ad8-9cc9142d12d0","Type":"ContainerDied","Data":"414accb291de910b47a17b89016eed4be600c142606fff6de269ea0f69757017"}
Nov 03 08:24:59.889209 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:24:59.889186    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:25:00.021033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.020969    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-tmp-dir\") pod \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") "
Nov 03 08:25:00.021227 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.021039    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-tls-certs\" (UniqueName: \"kubernetes.io/secret/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-tls-certs\") pod \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") "
Nov 03 08:25:00.021306 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.021229    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-kube-api-access-6dk6r\" (UniqueName: \"kubernetes.io/projected/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-kube-api-access-6dk6r\") pod \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") "
Nov 03 08:25:00.021306 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.021263    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-home\" (UniqueName: \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-home\") pod \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") "
Nov 03 08:25:00.023015 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.021467    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-model-cache\") pod \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") "
Nov 03 08:25:00.023015 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.021565    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-dshm\") pod \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\" (UID: \"d172e36b-89d5-43d3-9ad8-9cc9142d12d0\") "
Nov 03 08:25:00.023015 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.021638    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-model-cache" pod "d172e36b-89d5-43d3-9ad8-9cc9142d12d0" (UID: "d172e36b-89d5-43d3-9ad8-9cc9142d12d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:00.023015 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.022149    5070 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-model-cache\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.023015 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.022709    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-home" pod "d172e36b-89d5-43d3-9ad8-9cc9142d12d0" (UID: "d172e36b-89d5-43d3-9ad8-9cc9142d12d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:00.023015 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.022970    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-tmp-dir" pod "d172e36b-89d5-43d3-9ad8-9cc9142d12d0" (UID: "d172e36b-89d5-43d3-9ad8-9cc9142d12d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:00.024674 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.024646    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-tls-certs" pod "d172e36b-89d5-43d3-9ad8-9cc9142d12d0" (UID: "d172e36b-89d5-43d3-9ad8-9cc9142d12d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:25:00.025151 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.025122    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-kube-api-access-6dk6r" pod "d172e36b-89d5-43d3-9ad8-9cc9142d12d0" (UID: "d172e36b-89d5-43d3-9ad8-9cc9142d12d0"). InnerVolumeSpecName "kube-api-access-6dk6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:25:00.026064 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.026029    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-dshm" pod "d172e36b-89d5-43d3-9ad8-9cc9142d12d0" (UID: "d172e36b-89d5-43d3-9ad8-9cc9142d12d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:00.031258 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.031229    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb961482-e401-47f5-8aff-f604b364d631" path="/var/lib/kubelet/pods/cb961482-e401-47f5-8aff-f604b364d631/volumes"
Nov 03 08:25:00.126851 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.126665    5070 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-tls-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.126851 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.126698    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dk6r\" (UniqueName: \"kubernetes.io/projected/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-kube-api-access-6dk6r\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.126851 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.126718    5070 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-home\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.126851 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.126731    5070 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-dshm\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.126851 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.126745    5070 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d172e36b-89d5-43d3-9ad8-9cc9142d12d0-tmp-dir\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:00.746544 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.746513    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"
Nov 03 08:25:00.746544 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.746537    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2" event={"ID":"d172e36b-89d5-43d3-9ad8-9cc9142d12d0","Type":"ContainerDied","Data":"3f2853b63d867d78f3f1b693b43c5ef47c0db7e43e7cdf9a575dcd07a13f9893"}
Nov 03 08:25:00.747058 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.746585    5070 scope.go:122] "RemoveContainer" containerID="414accb291de910b47a17b89016eed4be600c142606fff6de269ea0f69757017"
Nov 03 08:25:00.763723 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.763701    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"]
Nov 03 08:25:00.772526 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:00.772499    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-bd085d/distinct-w6-bd085d-kserve-6d8c6dc45b-t6qz2"]
Nov 03 08:25:01.968691 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:01.968667    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:25:02.032600 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.032563    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" path="/var/lib/kubelet/pods/d172e36b-89d5-43d3-9ad8-9cc9142d12d0/volumes"
Nov 03 08:25:02.048744 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.048715    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istiod-ca-cert\") pod \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") "
Nov 03 08:25:02.048926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.048780    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-envoy\") pod \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") "
Nov 03 08:25:02.048926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.048830    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-workload-socket\") pod \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") "
Nov 03 08:25:02.048926 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.048895    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-kube-api-access-4bnsb\" (UniqueName: \"kubernetes.io/projected/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-kube-api-access-4bnsb\") pod \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") "
Nov 03 08:25:02.049119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.048943    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-data\") pod \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") "
Nov 03 08:25:02.049119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.048975    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-credential-socket\") pod \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") "
Nov 03 08:25:02.049119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.049013    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-token\" (UniqueName: \"kubernetes.io/projected/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-token\") pod \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") "
Nov 03 08:25:02.049119 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.049094    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istiod-ca-cert" pod "bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" (UID: "bbb0b5e6-9abd-45f5-b444-572bd1a53fc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:25:02.049331 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.049206    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-workload-socket" pod "bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" (UID: "bbb0b5e6-9abd-45f5-b444-572bd1a53fc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:02.049331 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.049236    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-workload-certs\") pod \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") "
Nov 03 08:25:02.049331 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.049297    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-podinfo\") pod \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\" (UID: \"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1\") "
Nov 03 08:25:02.049555 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.049536    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-workload-certs" pod "bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" (UID: "bbb0b5e6-9abd-45f5-b444-572bd1a53fc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:02.049619 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.049574    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-data" pod "bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" (UID: "bbb0b5e6-9abd-45f5-b444-572bd1a53fc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:02.049889 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.049868    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:02.049948 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.049903    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:02.049948 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.049920    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:02.049948 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.049934    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:02.049948 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.049871    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-credential-socket" pod "bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" (UID: "bbb0b5e6-9abd-45f5-b444-572bd1a53fc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:02.051262 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.051242    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-kube-api-access-4bnsb" pod "bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" (UID: "bbb0b5e6-9abd-45f5-b444-572bd1a53fc1"). InnerVolumeSpecName "kube-api-access-4bnsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:25:02.051525 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.051503    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-envoy" pod "bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" (UID: "bbb0b5e6-9abd-45f5-b444-572bd1a53fc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:02.051585 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.051521    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-token" pod "bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" (UID: "bbb0b5e6-9abd-45f5-b444-572bd1a53fc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:25:02.052023 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.052005    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-podinfo" pod "bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" (UID: "bbb0b5e6-9abd-45f5-b444-572bd1a53fc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:25:02.151554 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.151522    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bnsb\" (UniqueName: \"kubernetes.io/projected/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-kube-api-access-4bnsb\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:02.151554 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.151556    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:02.151722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.151571    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:02.151722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.151585    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:02.151722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.151598    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:02.766220 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.766186    5070 generic.go:356] "Generic (PLEG): container finished" podID="bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" containerID="b1401b6134a3ead2bdae6e4103b8040456f68f7acc759b0a14112e64ab659688" exitCode=0
Nov 03 08:25:02.766405 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.766224    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"
Nov 03 08:25:02.766405 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.766269    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89" event={"ID":"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1","Type":"ContainerDied","Data":"b1401b6134a3ead2bdae6e4103b8040456f68f7acc759b0a14112e64ab659688"}
Nov 03 08:25:02.766405 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.766325    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89" event={"ID":"bbb0b5e6-9abd-45f5-b444-572bd1a53fc1","Type":"ContainerDied","Data":"21e91b281c1fba8ad94876fe3dcd1ac0078ca33e256cf151d5a437faef0778a9"}
Nov 03 08:25:02.766405 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.766354    5070 scope.go:122] "RemoveContainer" containerID="b1401b6134a3ead2bdae6e4103b8040456f68f7acc759b0a14112e64ab659688"
Nov 03 08:25:02.778811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.778789    5070 scope.go:122] "RemoveContainer" containerID="b1401b6134a3ead2bdae6e4103b8040456f68f7acc759b0a14112e64ab659688"
Nov 03 08:25:02.779077 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:25:02.779059    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1401b6134a3ead2bdae6e4103b8040456f68f7acc759b0a14112e64ab659688\": container with ID starting with b1401b6134a3ead2bdae6e4103b8040456f68f7acc759b0a14112e64ab659688 not found: ID does not exist" containerID="b1401b6134a3ead2bdae6e4103b8040456f68f7acc759b0a14112e64ab659688"
Nov 03 08:25:02.779160 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.779083    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1401b6134a3ead2bdae6e4103b8040456f68f7acc759b0a14112e64ab659688"} err="failed to get container status \"b1401b6134a3ead2bdae6e4103b8040456f68f7acc759b0a14112e64ab659688\": rpc error: code = NotFound desc = could not find container \"b1401b6134a3ead2bdae6e4103b8040456f68f7acc759b0a14112e64ab659688\": container with ID starting with b1401b6134a3ead2bdae6e4103b8040456f68f7acc759b0a14112e64ab659688 not found: ID does not exist"
Nov 03 08:25:02.789368 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.789345    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"]
Nov 03 08:25:02.792994 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:02.792974    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-3e88b7-openshift-default-86b55fc97-lwb89"]
Nov 03 08:25:04.026909 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:04.026881    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" path="/var/lib/kubelet/pods/bbb0b5e6-9abd-45f5-b444-572bd1a53fc1/volumes"
Nov 03 08:25:08.029120 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:08.029079    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:25:11.151736 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:11.151706    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4"]
Nov 03 08:25:11.157271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:11.157247    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-bd085d-gnvt4"]
Nov 03 08:25:12.025626 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:12.025595    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfb66350-fefe-4b07-bd2c-f633a0bbfee3" path="/var/lib/kubelet/pods/bfb66350-fefe-4b07-bd2c-f633a0bbfee3/volumes"
Nov 03 08:25:14.272990 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:14.272962    5070 scope.go:122] "RemoveContainer" containerID="72f8580ef1bf93779927eefa63c21d948f333a1c29ffa692d2f49778d54e5e16"
Nov 03 08:25:18.136505 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:18.136470    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"]
Nov 03 08:25:18.137325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:18.137288    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s" podUID="f76ed594-a720-4470-85da-02885a26dd4c" containerName="istio-proxy" containerID="cri-o://9246caf5c83ab92efbfc6b529ad8a1e82cc5ceac0aa3bb0553d22249facf45c3" gracePeriod=30
Nov 03 08:25:18.151440 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:18.151405    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"]
Nov 03 08:25:18.151661 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:18.151633    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn" podUID="4fc257d6-34f4-412a-80c4-9a7d4462d2f9" containerName="istio-proxy" containerID="cri-o://2f031fbd4e1e1df8facbdd8a5355bb69211f9a7fa837238cbdf0f9b26cee72aa" gracePeriod=30
Nov 03 08:25:19.542832 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:19.542795    5070 patch_prober.go:29] interesting pod/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:25:19.543226 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:19.542851    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn" podUID="4fc257d6-34f4-412a-80c4-9a7d4462d2f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:25:22.053746 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:25:22.053712    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:25:23.449360 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.449328    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:25:23.496361 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.496332    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:25:23.577068 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.577035    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f76ed594-a720-4470-85da-02885a26dd4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f76ed594-a720-4470-85da-02885a26dd4c-istio-podinfo\") pod \"f76ed594-a720-4470-85da-02885a26dd4c\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") "
Nov 03 08:25:23.577300 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.577152    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f76ed594-a720-4470-85da-02885a26dd4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f76ed594-a720-4470-85da-02885a26dd4c-istiod-ca-cert\") pod \"f76ed594-a720-4470-85da-02885a26dd4c\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") "
Nov 03 08:25:23.577300 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.577262    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-istio-data\") pod \"f76ed594-a720-4470-85da-02885a26dd4c\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") "
Nov 03 08:25:23.577300 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.577298    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-token\" (UniqueName: \"kubernetes.io/projected/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-token\") pod \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") "
Nov 03 08:25:23.577483 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.577388    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-workload-certs\") pod \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") "
Nov 03 08:25:23.577550 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.577497    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istiod-ca-cert\") pod \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") "
Nov 03 08:25:23.577550 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.577536    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-credential-socket\") pod \"f76ed594-a720-4470-85da-02885a26dd4c\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") "
Nov 03 08:25:23.577670 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.577574    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-istio-envoy\") pod \"f76ed594-a720-4470-85da-02885a26dd4c\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") "
Nov 03 08:25:23.577670 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.577607    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f76ed594-a720-4470-85da-02885a26dd4c-kube-api-access-vj8sq\" (UniqueName: \"kubernetes.io/projected/f76ed594-a720-4470-85da-02885a26dd4c-kube-api-access-vj8sq\") pod \"f76ed594-a720-4470-85da-02885a26dd4c\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") "
Nov 03 08:25:23.577670 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.577651    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-envoy\") pod \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") "
Nov 03 08:25:23.577864 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.577680    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f76ed594-a720-4470-85da-02885a26dd4c-istio-token\" (UniqueName: \"kubernetes.io/projected/f76ed594-a720-4470-85da-02885a26dd4c-istio-token\") pod \"f76ed594-a720-4470-85da-02885a26dd4c\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") "
Nov 03 08:25:23.577864 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.577716    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-kube-api-access-q42g5\" (UniqueName: \"kubernetes.io/projected/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-kube-api-access-q42g5\") pod \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") "
Nov 03 08:25:23.577864 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.577788    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-podinfo\") pod \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") "
Nov 03 08:25:23.577864 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.577854    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-credential-socket\") pod \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") "
Nov 03 08:25:23.578077 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.577900    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-workload-socket\") pod \"f76ed594-a720-4470-85da-02885a26dd4c\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") "
Nov 03 08:25:23.578077 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.577961    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-data\") pod \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") "
Nov 03 08:25:23.578077 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.578005    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-workload-socket\") pod \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\" (UID: \"4fc257d6-34f4-412a-80c4-9a7d4462d2f9\") "
Nov 03 08:25:23.578077 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.578044    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-workload-certs\") pod \"f76ed594-a720-4470-85da-02885a26dd4c\" (UID: \"f76ed594-a720-4470-85da-02885a26dd4c\") "
Nov 03 08:25:23.578970 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.578906    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-workload-certs" pod "4fc257d6-34f4-412a-80c4-9a7d4462d2f9" (UID: "4fc257d6-34f4-412a-80c4-9a7d4462d2f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:23.579884 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.579841    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-data" pod "4fc257d6-34f4-412a-80c4-9a7d4462d2f9" (UID: "4fc257d6-34f4-412a-80c4-9a7d4462d2f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:23.580711 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.580675    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-workload-socket" pod "4fc257d6-34f4-412a-80c4-9a7d4462d2f9" (UID: "4fc257d6-34f4-412a-80c4-9a7d4462d2f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:23.580934 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.580911    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f76ed594-a720-4470-85da-02885a26dd4c-istio-podinfo" pod "f76ed594-a720-4470-85da-02885a26dd4c" (UID: "f76ed594-a720-4470-85da-02885a26dd4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:25:23.581074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.580967    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-credential-socket" pod "4fc257d6-34f4-412a-80c4-9a7d4462d2f9" (UID: "4fc257d6-34f4-412a-80c4-9a7d4462d2f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:23.581212 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.581176    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76ed594-a720-4470-85da-02885a26dd4c-istiod-ca-cert" pod "f76ed594-a720-4470-85da-02885a26dd4c" (UID: "f76ed594-a720-4470-85da-02885a26dd4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:25:23.581481 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.581450    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istiod-ca-cert" pod "4fc257d6-34f4-412a-80c4-9a7d4462d2f9" (UID: "4fc257d6-34f4-412a-80c4-9a7d4462d2f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:25:23.582209 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.582178    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-credential-socket" pod "f76ed594-a720-4470-85da-02885a26dd4c" (UID: "f76ed594-a720-4470-85da-02885a26dd4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:23.582209 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.582137    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-istio-data" pod "f76ed594-a720-4470-85da-02885a26dd4c" (UID: "f76ed594-a720-4470-85da-02885a26dd4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:23.582372 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.582243    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-workload-socket" pod "f76ed594-a720-4470-85da-02885a26dd4c" (UID: "f76ed594-a720-4470-85da-02885a26dd4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:23.582451 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.582423    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-workload-certs" pod "f76ed594-a720-4470-85da-02885a26dd4c" (UID: "f76ed594-a720-4470-85da-02885a26dd4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:23.583276 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.583251    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-kube-api-access-q42g5" pod "4fc257d6-34f4-412a-80c4-9a7d4462d2f9" (UID: "4fc257d6-34f4-412a-80c4-9a7d4462d2f9"). InnerVolumeSpecName "kube-api-access-q42g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:25:23.584485 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.584455    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-envoy" pod "4fc257d6-34f4-412a-80c4-9a7d4462d2f9" (UID: "4fc257d6-34f4-412a-80c4-9a7d4462d2f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:23.584580 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.584477    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-podinfo" pod "4fc257d6-34f4-412a-80c4-9a7d4462d2f9" (UID: "4fc257d6-34f4-412a-80c4-9a7d4462d2f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:25:23.584580 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.584497    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-token" pod "4fc257d6-34f4-412a-80c4-9a7d4462d2f9" (UID: "4fc257d6-34f4-412a-80c4-9a7d4462d2f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:25:23.585177 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.585069    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76ed594-a720-4470-85da-02885a26dd4c-kube-api-access-vj8sq" pod "f76ed594-a720-4470-85da-02885a26dd4c" (UID: "f76ed594-a720-4470-85da-02885a26dd4c"). InnerVolumeSpecName "kube-api-access-vj8sq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:25:23.585177 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.585130    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76ed594-a720-4470-85da-02885a26dd4c-istio-token" pod "f76ed594-a720-4470-85da-02885a26dd4c" (UID: "f76ed594-a720-4470-85da-02885a26dd4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:25:23.585324 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.585268    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-istio-envoy" pod "f76ed594-a720-4470-85da-02885a26dd4c" (UID: "f76ed594-a720-4470-85da-02885a26dd4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:25:23.680250 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.680194    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f76ed594-a720-4470-85da-02885a26dd4c-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.680250 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.680218    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f76ed594-a720-4470-85da-02885a26dd4c-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.680250 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.680229    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.680250 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.680237    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.680250 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.680246    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.680469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.680255    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.680469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.680264    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.680469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.680272    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.680469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.680281    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj8sq\" (UniqueName: \"kubernetes.io/projected/f76ed594-a720-4470-85da-02885a26dd4c-kube-api-access-vj8sq\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.680469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.680290    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.680469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.680298    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f76ed594-a720-4470-85da-02885a26dd4c-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.680469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.680308    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q42g5\" (UniqueName: \"kubernetes.io/projected/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-kube-api-access-q42g5\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.680469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.680316    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.680469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.680325    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.680469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.680335    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.680469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.680342    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.680469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.680350    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc257d6-34f4-412a-80c4-9a7d4462d2f9-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.680469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.680359    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f76ed594-a720-4470-85da-02885a26dd4c-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:25:23.982908 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.982824    5070 generic.go:356] "Generic (PLEG): container finished" podID="f76ed594-a720-4470-85da-02885a26dd4c" containerID="9246caf5c83ab92efbfc6b529ad8a1e82cc5ceac0aa3bb0553d22249facf45c3" exitCode=0
Nov 03 08:25:23.982908 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.982873    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"
Nov 03 08:25:23.983105 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.982909    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s" event={"ID":"f76ed594-a720-4470-85da-02885a26dd4c","Type":"ContainerDied","Data":"9246caf5c83ab92efbfc6b529ad8a1e82cc5ceac0aa3bb0553d22249facf45c3"}
Nov 03 08:25:23.983105 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.982989    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s" event={"ID":"f76ed594-a720-4470-85da-02885a26dd4c","Type":"ContainerDied","Data":"6289c0a2291092624b1bcb66ff3d67c84dc4d7b7b2702abdcda746969321a91d"}
Nov 03 08:25:23.983105 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.983011    5070 scope.go:122] "RemoveContainer" containerID="9246caf5c83ab92efbfc6b529ad8a1e82cc5ceac0aa3bb0553d22249facf45c3"
Nov 03 08:25:23.984937 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.984910    5070 generic.go:356] "Generic (PLEG): container finished" podID="4fc257d6-34f4-412a-80c4-9a7d4462d2f9" containerID="2f031fbd4e1e1df8facbdd8a5355bb69211f9a7fa837238cbdf0f9b26cee72aa" exitCode=0
Nov 03 08:25:23.985080 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.984948    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"
Nov 03 08:25:23.985080 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.984991    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn" event={"ID":"4fc257d6-34f4-412a-80c4-9a7d4462d2f9","Type":"ContainerDied","Data":"2f031fbd4e1e1df8facbdd8a5355bb69211f9a7fa837238cbdf0f9b26cee72aa"}
Nov 03 08:25:23.985080 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.985046    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn" event={"ID":"4fc257d6-34f4-412a-80c4-9a7d4462d2f9","Type":"ContainerDied","Data":"b67f45a4d154e302a94f306d0d1e175cb90876a43b232abaf52926f3edae9a04"}
Nov 03 08:25:23.995783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.995741    5070 scope.go:122] "RemoveContainer" containerID="9246caf5c83ab92efbfc6b529ad8a1e82cc5ceac0aa3bb0553d22249facf45c3"
Nov 03 08:25:23.996083 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:25:23.996058    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9246caf5c83ab92efbfc6b529ad8a1e82cc5ceac0aa3bb0553d22249facf45c3\": container with ID starting with 9246caf5c83ab92efbfc6b529ad8a1e82cc5ceac0aa3bb0553d22249facf45c3 not found: ID does not exist" containerID="9246caf5c83ab92efbfc6b529ad8a1e82cc5ceac0aa3bb0553d22249facf45c3"
Nov 03 08:25:23.996167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.996090    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9246caf5c83ab92efbfc6b529ad8a1e82cc5ceac0aa3bb0553d22249facf45c3"} err="failed to get container status \"9246caf5c83ab92efbfc6b529ad8a1e82cc5ceac0aa3bb0553d22249facf45c3\": rpc error: code = NotFound desc = could not find container \"9246caf5c83ab92efbfc6b529ad8a1e82cc5ceac0aa3bb0553d22249facf45c3\": container with ID starting with 9246caf5c83ab92efbfc6b529ad8a1e82cc5ceac0aa3bb0553d22249facf45c3 not found: ID does not exist"
Nov 03 08:25:23.996167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:23.996108    5070 scope.go:122] "RemoveContainer" containerID="2f031fbd4e1e1df8facbdd8a5355bb69211f9a7fa837238cbdf0f9b26cee72aa"
Nov 03 08:25:24.007836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:24.007812    5070 scope.go:122] "RemoveContainer" containerID="ac1ef05fa603dd69fa4d8d1d16e234f61492af355d24860a2e6e264a6cb2ed44"
Nov 03 08:25:24.010368 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:24.010345    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"]
Nov 03 08:25:24.015225 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:24.015205    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-2zf9s"]
Nov 03 08:25:24.019943 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:24.019926    5070 scope.go:122] "RemoveContainer" containerID="2f031fbd4e1e1df8facbdd8a5355bb69211f9a7fa837238cbdf0f9b26cee72aa"
Nov 03 08:25:24.020241 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:25:24.020219    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f031fbd4e1e1df8facbdd8a5355bb69211f9a7fa837238cbdf0f9b26cee72aa\": container with ID starting with 2f031fbd4e1e1df8facbdd8a5355bb69211f9a7fa837238cbdf0f9b26cee72aa not found: ID does not exist" containerID="2f031fbd4e1e1df8facbdd8a5355bb69211f9a7fa837238cbdf0f9b26cee72aa"
Nov 03 08:25:24.020305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:24.020249    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f031fbd4e1e1df8facbdd8a5355bb69211f9a7fa837238cbdf0f9b26cee72aa"} err="failed to get container status \"2f031fbd4e1e1df8facbdd8a5355bb69211f9a7fa837238cbdf0f9b26cee72aa\": rpc error: code = NotFound desc = could not find container \"2f031fbd4e1e1df8facbdd8a5355bb69211f9a7fa837238cbdf0f9b26cee72aa\": container with ID starting with 2f031fbd4e1e1df8facbdd8a5355bb69211f9a7fa837238cbdf0f9b26cee72aa not found: ID does not exist"
Nov 03 08:25:24.020305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:24.020266    5070 scope.go:122] "RemoveContainer" containerID="ac1ef05fa603dd69fa4d8d1d16e234f61492af355d24860a2e6e264a6cb2ed44"
Nov 03 08:25:24.020527 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:25:24.020503    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1ef05fa603dd69fa4d8d1d16e234f61492af355d24860a2e6e264a6cb2ed44\": container with ID starting with ac1ef05fa603dd69fa4d8d1d16e234f61492af355d24860a2e6e264a6cb2ed44 not found: ID does not exist" containerID="ac1ef05fa603dd69fa4d8d1d16e234f61492af355d24860a2e6e264a6cb2ed44"
Nov 03 08:25:24.020639 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:24.020531    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1ef05fa603dd69fa4d8d1d16e234f61492af355d24860a2e6e264a6cb2ed44"} err="failed to get container status \"ac1ef05fa603dd69fa4d8d1d16e234f61492af355d24860a2e6e264a6cb2ed44\": rpc error: code = NotFound desc = could not find container \"ac1ef05fa603dd69fa4d8d1d16e234f61492af355d24860a2e6e264a6cb2ed44\": container with ID starting with ac1ef05fa603dd69fa4d8d1d16e234f61492af355d24860a2e6e264a6cb2ed44 not found: ID does not exist"
Nov 03 08:25:24.027580 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:24.027559    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f76ed594-a720-4470-85da-02885a26dd4c" path="/var/lib/kubelet/pods/f76ed594-a720-4470-85da-02885a26dd4c/volumes"
Nov 03 08:25:24.027904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:24.027892    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"]
Nov 03 08:25:24.028118 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:24.028101    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-bd085d-openshift-default-5d55b95c87-f7gmn"]
Nov 03 08:25:26.025143 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:26.025107    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fc257d6-34f4-412a-80c4-9a7d4462d2f9" path="/var/lib/kubelet/pods/4fc257d6-34f4-412a-80c4-9a7d4462d2f9/volumes"
Nov 03 08:25:48.672500 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:25:48.672416    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:25:52.052034 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:25:52.052002    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:26:08.186408 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:08.186375    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:26:18.139805 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:18.139768    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"]
Nov 03 08:26:18.140237 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:18.139965    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r" podUID="8615eb5b-41a4-4c01-b550-c1d3df743738" containerName="istio-proxy" containerID="cri-o://48d010e07ea12dd9524c8345df22b4be0189b2c78fbe3bd1151814c129eeeadf" gracePeriod=30
Nov 03 08:26:21.012141 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.012101    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb3a58d0-ql8t6"]
Nov 03 08:26:21.012726 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.012707    5070 cpu_manager.go:414] "containerMap: removing container" podUID="bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" containerName="istio-proxy"
Nov 03 08:26:21.012853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.012727    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" containerName="istio-proxy"
Nov 03 08:26:21.012853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.012746    5070 cpu_manager.go:414] "containerMap: removing container" podUID="42228923-0917-4753-af54-c3fc849be091" containerName="revoke-keys"
Nov 03 08:26:21.012853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.012752    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42228923-0917-4753-af54-c3fc849be091" containerName="revoke-keys"
Nov 03 08:26:21.012853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.012793    5070 cpu_manager.go:414] "containerMap: removing container" podUID="4fc257d6-34f4-412a-80c4-9a7d4462d2f9" containerName="istio-proxy"
Nov 03 08:26:21.012853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.012799    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fc257d6-34f4-412a-80c4-9a7d4462d2f9" containerName="istio-proxy"
Nov 03 08:26:21.012853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.012807    5070 cpu_manager.go:414] "containerMap: removing container" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main"
Nov 03 08:26:21.012853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.012812    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main"
Nov 03 08:26:21.012853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.012829    5070 cpu_manager.go:414] "containerMap: removing container" podUID="4fc257d6-34f4-412a-80c4-9a7d4462d2f9" containerName="istio-proxy"
Nov 03 08:26:21.012853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.012834    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fc257d6-34f4-412a-80c4-9a7d4462d2f9" containerName="istio-proxy"
Nov 03 08:26:21.012853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.012840    5070 cpu_manager.go:414] "containerMap: removing container" podUID="cb961482-e401-47f5-8aff-f604b364d631" containerName="payload-processing"
Nov 03 08:26:21.012853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.012845    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb961482-e401-47f5-8aff-f604b364d631" containerName="payload-processing"
Nov 03 08:26:21.012853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.012852    5070 cpu_manager.go:414] "containerMap: removing container" podUID="506fb467-77cf-4b49-8e99-b787b98b490f" containerName="revoke-keys"
Nov 03 08:26:21.012853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.012857    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="506fb467-77cf-4b49-8e99-b787b98b490f" containerName="revoke-keys"
Nov 03 08:26:21.013330 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.012880    5070 cpu_manager.go:414] "containerMap: removing container" podUID="f76ed594-a720-4470-85da-02885a26dd4c" containerName="istio-proxy"
Nov 03 08:26:21.013330 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.012884    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f76ed594-a720-4470-85da-02885a26dd4c" containerName="istio-proxy"
Nov 03 08:26:21.013330 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.012981    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fc257d6-34f4-412a-80c4-9a7d4462d2f9" containerName="istio-proxy"
Nov 03 08:26:21.013330 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.012988    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb961482-e401-47f5-8aff-f604b364d631" containerName="payload-processing"
Nov 03 08:26:21.013330 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.012996    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="506fb467-77cf-4b49-8e99-b787b98b490f" containerName="revoke-keys"
Nov 03 08:26:21.013330 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.013006    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="f76ed594-a720-4470-85da-02885a26dd4c" containerName="istio-proxy"
Nov 03 08:26:21.013330 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.013014    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbb0b5e6-9abd-45f5-b444-572bd1a53fc1" containerName="istio-proxy"
Nov 03 08:26:21.013330 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.013022    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="42228923-0917-4753-af54-c3fc849be091" containerName="revoke-keys"
Nov 03 08:26:21.013330 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.013030    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fc257d6-34f4-412a-80c4-9a7d4462d2f9" containerName="istio-proxy"
Nov 03 08:26:21.013330 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.013035    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="d172e36b-89d5-43d3-9ad8-9cc9142d12d0" containerName="main"
Nov 03 08:26:21.016575 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.016558    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb3a58d0-ql8t6"
Nov 03 08:26:21.021938 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.021905    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:26:21.022160 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.022141    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:26:21.022160 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.022145    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:26:21.022803 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.022780    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:26:21.025387 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.025354    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb3a58d0-ql8t6"]
Nov 03 08:26:21.075871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.064125    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv45g\" (UniqueName: \"kubernetes.io/projected/3868d631-022d-4a08-b8aa-1d70aa1ebc01-kube-api-access-lv45g\") pod \"maas-api-revoke-sub-cb3a58d0-ql8t6\" (UID: \"3868d631-022d-4a08-b8aa-1d70aa1ebc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb3a58d0-ql8t6"
Nov 03 08:26:21.076422 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.076392    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3868d631-022d-4a08-b8aa-1d70aa1ebc01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb3a58d0-ql8t6\" (UID: \"3868d631-022d-4a08-b8aa-1d70aa1ebc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb3a58d0-ql8t6"
Nov 03 08:26:21.076729 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.076674    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3868d631-022d-4a08-b8aa-1d70aa1ebc01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb3a58d0-ql8t6\" (UID: \"3868d631-022d-4a08-b8aa-1d70aa1ebc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb3a58d0-ql8t6"
Nov 03 08:26:21.178469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.178434    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv45g\" (UniqueName: \"kubernetes.io/projected/3868d631-022d-4a08-b8aa-1d70aa1ebc01-kube-api-access-lv45g\") pod \"maas-api-revoke-sub-cb3a58d0-ql8t6\" (UID: \"3868d631-022d-4a08-b8aa-1d70aa1ebc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb3a58d0-ql8t6"
Nov 03 08:26:21.178645 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.178513    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3868d631-022d-4a08-b8aa-1d70aa1ebc01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb3a58d0-ql8t6\" (UID: \"3868d631-022d-4a08-b8aa-1d70aa1ebc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb3a58d0-ql8t6"
Nov 03 08:26:21.178645 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.178539    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3868d631-022d-4a08-b8aa-1d70aa1ebc01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb3a58d0-ql8t6\" (UID: \"3868d631-022d-4a08-b8aa-1d70aa1ebc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb3a58d0-ql8t6"
Nov 03 08:26:21.179161 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.179138    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3868d631-022d-4a08-b8aa-1d70aa1ebc01-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb3a58d0-ql8t6\" (UID: \"3868d631-022d-4a08-b8aa-1d70aa1ebc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb3a58d0-ql8t6"
Nov 03 08:26:21.181077 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.181046    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3868d631-022d-4a08-b8aa-1d70aa1ebc01-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb3a58d0-ql8t6\" (UID: \"3868d631-022d-4a08-b8aa-1d70aa1ebc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb3a58d0-ql8t6"
Nov 03 08:26:21.185038 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.185020    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv45g\" (UniqueName: \"kubernetes.io/projected/3868d631-022d-4a08-b8aa-1d70aa1ebc01-kube-api-access-lv45g\") pod \"maas-api-revoke-sub-cb3a58d0-ql8t6\" (UID: \"3868d631-022d-4a08-b8aa-1d70aa1ebc01\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb3a58d0-ql8t6"
Nov 03 08:26:21.332106 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.332052    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb3a58d0-ql8t6"
Nov 03 08:26:21.610333 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:21.610309    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb3a58d0-ql8t6"]
Nov 03 08:26:21.611177 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:26:21.611150    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3868d631_022d_4a08_b8aa_1d70aa1ebc01.slice/crio-7a3d1e638be6d5baa9d603310eb4e2c8c4630a31f1bd389e7ce7a2f91764f58c WatchSource:0}: Error finding container 7a3d1e638be6d5baa9d603310eb4e2c8c4630a31f1bd389e7ce7a2f91764f58c: Status 404 returned error can't find the container with id 7a3d1e638be6d5baa9d603310eb4e2c8c4630a31f1bd389e7ce7a2f91764f58c
Nov 03 08:26:22.051775 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:26:22.051728    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:26:22.459615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:22.459581    5070 generic.go:356] "Generic (PLEG): container finished" podID="3868d631-022d-4a08-b8aa-1d70aa1ebc01" containerID="37143d286f2ad40097d44c27f85e43ad89e54af3d7b94137a6662c74d825882a" exitCode=0
Nov 03 08:26:22.459778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:22.459663    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb3a58d0-ql8t6" event={"ID":"3868d631-022d-4a08-b8aa-1d70aa1ebc01","Type":"ContainerDied","Data":"37143d286f2ad40097d44c27f85e43ad89e54af3d7b94137a6662c74d825882a"}
Nov 03 08:26:22.459778 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:22.459715    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb3a58d0-ql8t6" event={"ID":"3868d631-022d-4a08-b8aa-1d70aa1ebc01","Type":"ContainerStarted","Data":"7a3d1e638be6d5baa9d603310eb4e2c8c4630a31f1bd389e7ce7a2f91764f58c"}
Nov 03 08:26:23.407471 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.407411    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:26:23.470040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.470006    5070 generic.go:356] "Generic (PLEG): container finished" podID="8615eb5b-41a4-4c01-b550-c1d3df743738" containerID="48d010e07ea12dd9524c8345df22b4be0189b2c78fbe3bd1151814c129eeeadf" exitCode=0
Nov 03 08:26:23.470216 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.470058    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"
Nov 03 08:26:23.470216 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.470098    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r" event={"ID":"8615eb5b-41a4-4c01-b550-c1d3df743738","Type":"ContainerDied","Data":"48d010e07ea12dd9524c8345df22b4be0189b2c78fbe3bd1151814c129eeeadf"}
Nov 03 08:26:23.470354 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.470221    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r" event={"ID":"8615eb5b-41a4-4c01-b550-c1d3df743738","Type":"ContainerDied","Data":"7274782777fa4786380feac996e280a4843abbd42cd6d052018a4219e1394a5c"}
Nov 03 08:26:23.470354 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.470260    5070 scope.go:122] "RemoveContainer" containerID="48d010e07ea12dd9524c8345df22b4be0189b2c78fbe3bd1151814c129eeeadf"
Nov 03 08:26:23.486870 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.486845    5070 scope.go:122] "RemoveContainer" containerID="48d010e07ea12dd9524c8345df22b4be0189b2c78fbe3bd1151814c129eeeadf"
Nov 03 08:26:23.487309 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:26:23.487278    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d010e07ea12dd9524c8345df22b4be0189b2c78fbe3bd1151814c129eeeadf\": container with ID starting with 48d010e07ea12dd9524c8345df22b4be0189b2c78fbe3bd1151814c129eeeadf not found: ID does not exist" containerID="48d010e07ea12dd9524c8345df22b4be0189b2c78fbe3bd1151814c129eeeadf"
Nov 03 08:26:23.487457 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.487317    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d010e07ea12dd9524c8345df22b4be0189b2c78fbe3bd1151814c129eeeadf"} err="failed to get container status \"48d010e07ea12dd9524c8345df22b4be0189b2c78fbe3bd1151814c129eeeadf\": rpc error: code = NotFound desc = could not find container \"48d010e07ea12dd9524c8345df22b4be0189b2c78fbe3bd1151814c129eeeadf\": container with ID starting with 48d010e07ea12dd9524c8345df22b4be0189b2c78fbe3bd1151814c129eeeadf not found: ID does not exist"
Nov 03 08:26:23.512506 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.512481    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-credential-socket\") pod \"8615eb5b-41a4-4c01-b550-c1d3df743738\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") "
Nov 03 08:26:23.512685 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.512518    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-workload-certs\") pod \"8615eb5b-41a4-4c01-b550-c1d3df743738\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") "
Nov 03 08:26:23.512685 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.512561    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-workload-socket\") pod \"8615eb5b-41a4-4c01-b550-c1d3df743738\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") "
Nov 03 08:26:23.512685 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.512640    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-token\" (UniqueName: \"kubernetes.io/projected/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-token\") pod \"8615eb5b-41a4-4c01-b550-c1d3df743738\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") "
Nov 03 08:26:23.512685 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.512657    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8615eb5b-41a4-4c01-b550-c1d3df743738-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8615eb5b-41a4-4c01-b550-c1d3df743738-istiod-ca-cert\") pod \"8615eb5b-41a4-4c01-b550-c1d3df743738\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") "
Nov 03 08:26:23.512964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.512748    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-envoy\") pod \"8615eb5b-41a4-4c01-b550-c1d3df743738\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") "
Nov 03 08:26:23.512964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.512751    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-credential-socket" pod "8615eb5b-41a4-4c01-b550-c1d3df743738" (UID: "8615eb5b-41a4-4c01-b550-c1d3df743738"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:26:23.512964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.512789    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8615eb5b-41a4-4c01-b550-c1d3df743738-kube-api-access-j749l\" (UniqueName: \"kubernetes.io/projected/8615eb5b-41a4-4c01-b550-c1d3df743738-kube-api-access-j749l\") pod \"8615eb5b-41a4-4c01-b550-c1d3df743738\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") "
Nov 03 08:26:23.513127 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.512965    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-data\") pod \"8615eb5b-41a4-4c01-b550-c1d3df743738\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") "
Nov 03 08:26:23.513127 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.512963    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-workload-certs" pod "8615eb5b-41a4-4c01-b550-c1d3df743738" (UID: "8615eb5b-41a4-4c01-b550-c1d3df743738"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:26:23.513127 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.513062    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-podinfo\") pod \"8615eb5b-41a4-4c01-b550-c1d3df743738\" (UID: \"8615eb5b-41a4-4c01-b550-c1d3df743738\") "
Nov 03 08:26:23.513294 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.513140    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8615eb5b-41a4-4c01-b550-c1d3df743738-istiod-ca-cert" pod "8615eb5b-41a4-4c01-b550-c1d3df743738" (UID: "8615eb5b-41a4-4c01-b550-c1d3df743738"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:26:23.513294 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.513263    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-workload-socket" pod "8615eb5b-41a4-4c01-b550-c1d3df743738" (UID: "8615eb5b-41a4-4c01-b550-c1d3df743738"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:26:23.513401 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.513309    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-data" pod "8615eb5b-41a4-4c01-b550-c1d3df743738" (UID: "8615eb5b-41a4-4c01-b550-c1d3df743738"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:26:23.513975 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.513950    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:23.513975 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.513973    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:23.514167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.513982    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:23.514167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.513991    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8615eb5b-41a4-4c01-b550-c1d3df743738-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:23.514167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.514000    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:23.515579 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.515541    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-podinfo" pod "8615eb5b-41a4-4c01-b550-c1d3df743738" (UID: "8615eb5b-41a4-4c01-b550-c1d3df743738"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:26:23.515863 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.515841    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8615eb5b-41a4-4c01-b550-c1d3df743738-kube-api-access-j749l" pod "8615eb5b-41a4-4c01-b550-c1d3df743738" (UID: "8615eb5b-41a4-4c01-b550-c1d3df743738"). InnerVolumeSpecName "kube-api-access-j749l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:26:23.516276 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.516242    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-token" pod "8615eb5b-41a4-4c01-b550-c1d3df743738" (UID: "8615eb5b-41a4-4c01-b550-c1d3df743738"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:26:23.516864 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.516834    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-envoy" pod "8615eb5b-41a4-4c01-b550-c1d3df743738" (UID: "8615eb5b-41a4-4c01-b550-c1d3df743738"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:26:23.614741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.614712    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:23.614741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.614739    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j749l\" (UniqueName: \"kubernetes.io/projected/8615eb5b-41a4-4c01-b550-c1d3df743738-kube-api-access-j749l\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:23.614931 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.614750    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:23.614931 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.614812    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8615eb5b-41a4-4c01-b550-c1d3df743738-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:23.634171 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.634147    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb3a58d0-ql8t6"
Nov 03 08:26:23.715365 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.715300    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3868d631-022d-4a08-b8aa-1d70aa1ebc01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3868d631-022d-4a08-b8aa-1d70aa1ebc01-maas-api-openshift-ca\") pod \"3868d631-022d-4a08-b8aa-1d70aa1ebc01\" (UID: \"3868d631-022d-4a08-b8aa-1d70aa1ebc01\") "
Nov 03 08:26:23.715365 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.715358    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3868d631-022d-4a08-b8aa-1d70aa1ebc01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3868d631-022d-4a08-b8aa-1d70aa1ebc01-maas-api-xks-ca\") pod \"3868d631-022d-4a08-b8aa-1d70aa1ebc01\" (UID: \"3868d631-022d-4a08-b8aa-1d70aa1ebc01\") "
Nov 03 08:26:23.715570 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.715544    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3868d631-022d-4a08-b8aa-1d70aa1ebc01-kube-api-access-lv45g\" (UniqueName: \"kubernetes.io/projected/3868d631-022d-4a08-b8aa-1d70aa1ebc01-kube-api-access-lv45g\") pod \"3868d631-022d-4a08-b8aa-1d70aa1ebc01\" (UID: \"3868d631-022d-4a08-b8aa-1d70aa1ebc01\") "
Nov 03 08:26:23.715717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.715695    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3868d631-022d-4a08-b8aa-1d70aa1ebc01-maas-api-openshift-ca" pod "3868d631-022d-4a08-b8aa-1d70aa1ebc01" (UID: "3868d631-022d-4a08-b8aa-1d70aa1ebc01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:26:23.716176 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.716155    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3868d631-022d-4a08-b8aa-1d70aa1ebc01-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:23.717410 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.717387    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3868d631-022d-4a08-b8aa-1d70aa1ebc01-maas-api-xks-ca" pod "3868d631-022d-4a08-b8aa-1d70aa1ebc01" (UID: "3868d631-022d-4a08-b8aa-1d70aa1ebc01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:26:23.717532 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.717512    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3868d631-022d-4a08-b8aa-1d70aa1ebc01-kube-api-access-lv45g" pod "3868d631-022d-4a08-b8aa-1d70aa1ebc01" (UID: "3868d631-022d-4a08-b8aa-1d70aa1ebc01"). InnerVolumeSpecName "kube-api-access-lv45g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:26:23.793895 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.793871    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"]
Nov 03 08:26:23.798000 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.797979    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-wpf5r"]
Nov 03 08:26:23.817473 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.817450    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3868d631-022d-4a08-b8aa-1d70aa1ebc01-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:23.817473 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:23.817473    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv45g\" (UniqueName: \"kubernetes.io/projected/3868d631-022d-4a08-b8aa-1d70aa1ebc01-kube-api-access-lv45g\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:24.027011 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:24.026942    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8615eb5b-41a4-4c01-b550-c1d3df743738" path="/var/lib/kubelet/pods/8615eb5b-41a4-4c01-b550-c1d3df743738/volumes"
Nov 03 08:26:24.484430 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:24.484402    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb3a58d0-ql8t6"
Nov 03 08:26:24.484819 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:24.484432    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb3a58d0-ql8t6" event={"ID":"3868d631-022d-4a08-b8aa-1d70aa1ebc01","Type":"ContainerDied","Data":"7a3d1e638be6d5baa9d603310eb4e2c8c4630a31f1bd389e7ce7a2f91764f58c"}
Nov 03 08:26:24.484819 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:24.484487    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a3d1e638be6d5baa9d603310eb4e2c8c4630a31f1bd389e7ce7a2f91764f58c"
Nov 03 08:26:27.727253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:27.727223    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:26:31.145394 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.145359    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh"]
Nov 03 08:26:31.145920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.145906    5070 cpu_manager.go:414] "containerMap: removing container" podUID="8615eb5b-41a4-4c01-b550-c1d3df743738" containerName="istio-proxy"
Nov 03 08:26:31.145920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.145920    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8615eb5b-41a4-4c01-b550-c1d3df743738" containerName="istio-proxy"
Nov 03 08:26:31.146006 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.145943    5070 cpu_manager.go:414] "containerMap: removing container" podUID="3868d631-022d-4a08-b8aa-1d70aa1ebc01" containerName="revoke-keys"
Nov 03 08:26:31.146006 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.145948    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3868d631-022d-4a08-b8aa-1d70aa1ebc01" containerName="revoke-keys"
Nov 03 08:26:31.146072 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.146067    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="3868d631-022d-4a08-b8aa-1d70aa1ebc01" containerName="revoke-keys"
Nov 03 08:26:31.146107 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.146081    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="8615eb5b-41a4-4c01-b550-c1d3df743738" containerName="istio-proxy"
Nov 03 08:26:31.150320 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.150299    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh"
Nov 03 08:26:31.154565 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.154541    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:26:31.154721 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.154548    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:26:31.154721 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.154556    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:26:31.154901 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.154604    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:26:31.159486 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.159463    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh"]
Nov 03 08:26:31.303049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.303018    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e2bb450-f2a0-41e2-90bb-9600e443f7b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-025362e9-4ghsh\" (UID: \"5e2bb450-f2a0-41e2-90bb-9600e443f7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh"
Nov 03 08:26:31.303215 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.303081    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2znzk\" (UniqueName: \"kubernetes.io/projected/5e2bb450-f2a0-41e2-90bb-9600e443f7b2-kube-api-access-2znzk\") pod \"maas-api-revoke-sub-025362e9-4ghsh\" (UID: \"5e2bb450-f2a0-41e2-90bb-9600e443f7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh"
Nov 03 08:26:31.303275 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.303226    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e2bb450-f2a0-41e2-90bb-9600e443f7b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-025362e9-4ghsh\" (UID: \"5e2bb450-f2a0-41e2-90bb-9600e443f7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh"
Nov 03 08:26:31.404894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.404823    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2znzk\" (UniqueName: \"kubernetes.io/projected/5e2bb450-f2a0-41e2-90bb-9600e443f7b2-kube-api-access-2znzk\") pod \"maas-api-revoke-sub-025362e9-4ghsh\" (UID: \"5e2bb450-f2a0-41e2-90bb-9600e443f7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh"
Nov 03 08:26:31.405044 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.404901    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e2bb450-f2a0-41e2-90bb-9600e443f7b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-025362e9-4ghsh\" (UID: \"5e2bb450-f2a0-41e2-90bb-9600e443f7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh"
Nov 03 08:26:31.405044 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.404970    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e2bb450-f2a0-41e2-90bb-9600e443f7b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-025362e9-4ghsh\" (UID: \"5e2bb450-f2a0-41e2-90bb-9600e443f7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh"
Nov 03 08:26:31.405672 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.405646    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e2bb450-f2a0-41e2-90bb-9600e443f7b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-025362e9-4ghsh\" (UID: \"5e2bb450-f2a0-41e2-90bb-9600e443f7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh"
Nov 03 08:26:31.407301 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.407276    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e2bb450-f2a0-41e2-90bb-9600e443f7b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-025362e9-4ghsh\" (UID: \"5e2bb450-f2a0-41e2-90bb-9600e443f7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh"
Nov 03 08:26:31.412350 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.412331    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2znzk\" (UniqueName: \"kubernetes.io/projected/5e2bb450-f2a0-41e2-90bb-9600e443f7b2-kube-api-access-2znzk\") pod \"maas-api-revoke-sub-025362e9-4ghsh\" (UID: \"5e2bb450-f2a0-41e2-90bb-9600e443f7b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh"
Nov 03 08:26:31.466094 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.466063    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh"
Nov 03 08:26:31.744670 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:31.744637    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh"]
Nov 03 08:26:31.745800 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:26:31.745747    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2bb450_f2a0_41e2_90bb_9600e443f7b2.slice/crio-f5faa3415138c5b32fac33a76223968f9c34d3b54a1f0365de7b1ec2ab918ca1 WatchSource:0}: Error finding container f5faa3415138c5b32fac33a76223968f9c34d3b54a1f0365de7b1ec2ab918ca1: Status 404 returned error can't find the container with id f5faa3415138c5b32fac33a76223968f9c34d3b54a1f0365de7b1ec2ab918ca1
Nov 03 08:26:32.564088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:32.564055    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh" event={"ID":"5e2bb450-f2a0-41e2-90bb-9600e443f7b2","Type":"ContainerStarted","Data":"6ef495fa6ceddb56d57d1480cf4661fdc7b8a24205c27e72324384f8b8f8ca9f"}
Nov 03 08:26:32.564571 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:32.564114    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh" event={"ID":"5e2bb450-f2a0-41e2-90bb-9600e443f7b2","Type":"ContainerStarted","Data":"f5faa3415138c5b32fac33a76223968f9c34d3b54a1f0365de7b1ec2ab918ca1"}
Nov 03 08:26:32.579150 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:32.579105    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh" podStartSLOduration=1.579091231 podStartE2EDuration="1.579091231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:26:32.576382806 +0000 UTC m=+1990.690846736" watchObservedRunningTime="2026-11-03 08:26:32.579091231 +0000 UTC m=+1990.693555162"
Nov 03 08:26:33.581171 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:33.581135    5070 generic.go:356] "Generic (PLEG): container finished" podID="5e2bb450-f2a0-41e2-90bb-9600e443f7b2" containerID="6ef495fa6ceddb56d57d1480cf4661fdc7b8a24205c27e72324384f8b8f8ca9f" exitCode=0
Nov 03 08:26:33.581638 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:33.581306    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh" event={"ID":"5e2bb450-f2a0-41e2-90bb-9600e443f7b2","Type":"ContainerDied","Data":"6ef495fa6ceddb56d57d1480cf4661fdc7b8a24205c27e72324384f8b8f8ca9f"}
Nov 03 08:26:34.309225 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.309191    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds"]
Nov 03 08:26:34.313482 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.313464    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds"
Nov 03 08:26:34.322002 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.321975    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds"]
Nov 03 08:26:34.340496 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.340469    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqts6\" (UniqueName: \"kubernetes.io/projected/18d88475-e8f2-42dc-b2c9-420e69c64636-kube-api-access-cqts6\") pod \"maas-api-revoke-sub-36111c31-2j4ds\" (UID: \"18d88475-e8f2-42dc-b2c9-420e69c64636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds"
Nov 03 08:26:34.340630 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.340537    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d88475-e8f2-42dc-b2c9-420e69c64636-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36111c31-2j4ds\" (UID: \"18d88475-e8f2-42dc-b2c9-420e69c64636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds"
Nov 03 08:26:34.340706 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.340680    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d88475-e8f2-42dc-b2c9-420e69c64636-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36111c31-2j4ds\" (UID: \"18d88475-e8f2-42dc-b2c9-420e69c64636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds"
Nov 03 08:26:34.441631 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.441605    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d88475-e8f2-42dc-b2c9-420e69c64636-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36111c31-2j4ds\" (UID: \"18d88475-e8f2-42dc-b2c9-420e69c64636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds"
Nov 03 08:26:34.441788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.441714    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqts6\" (UniqueName: \"kubernetes.io/projected/18d88475-e8f2-42dc-b2c9-420e69c64636-kube-api-access-cqts6\") pod \"maas-api-revoke-sub-36111c31-2j4ds\" (UID: \"18d88475-e8f2-42dc-b2c9-420e69c64636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds"
Nov 03 08:26:34.441849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.441817    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d88475-e8f2-42dc-b2c9-420e69c64636-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36111c31-2j4ds\" (UID: \"18d88475-e8f2-42dc-b2c9-420e69c64636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds"
Nov 03 08:26:34.442410 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.442386    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d88475-e8f2-42dc-b2c9-420e69c64636-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36111c31-2j4ds\" (UID: \"18d88475-e8f2-42dc-b2c9-420e69c64636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds"
Nov 03 08:26:34.443997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.443978    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d88475-e8f2-42dc-b2c9-420e69c64636-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36111c31-2j4ds\" (UID: \"18d88475-e8f2-42dc-b2c9-420e69c64636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds"
Nov 03 08:26:34.448418 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.448395    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqts6\" (UniqueName: \"kubernetes.io/projected/18d88475-e8f2-42dc-b2c9-420e69c64636-kube-api-access-cqts6\") pod \"maas-api-revoke-sub-36111c31-2j4ds\" (UID: \"18d88475-e8f2-42dc-b2c9-420e69c64636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds"
Nov 03 08:26:34.630277 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.630238    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds"
Nov 03 08:26:34.783684 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.783655    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh"
Nov 03 08:26:34.846898 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.846866    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e2bb450-f2a0-41e2-90bb-9600e443f7b2-kube-api-access-2znzk\" (UniqueName: \"kubernetes.io/projected/5e2bb450-f2a0-41e2-90bb-9600e443f7b2-kube-api-access-2znzk\") pod \"5e2bb450-f2a0-41e2-90bb-9600e443f7b2\" (UID: \"5e2bb450-f2a0-41e2-90bb-9600e443f7b2\") "
Nov 03 08:26:34.847060 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.846920    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e2bb450-f2a0-41e2-90bb-9600e443f7b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e2bb450-f2a0-41e2-90bb-9600e443f7b2-maas-api-openshift-ca\") pod \"5e2bb450-f2a0-41e2-90bb-9600e443f7b2\" (UID: \"5e2bb450-f2a0-41e2-90bb-9600e443f7b2\") "
Nov 03 08:26:34.847060 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.847053    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e2bb450-f2a0-41e2-90bb-9600e443f7b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e2bb450-f2a0-41e2-90bb-9600e443f7b2-maas-api-xks-ca\") pod \"5e2bb450-f2a0-41e2-90bb-9600e443f7b2\" (UID: \"5e2bb450-f2a0-41e2-90bb-9600e443f7b2\") "
Nov 03 08:26:34.847358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.847287    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e2bb450-f2a0-41e2-90bb-9600e443f7b2-maas-api-openshift-ca" pod "5e2bb450-f2a0-41e2-90bb-9600e443f7b2" (UID: "5e2bb450-f2a0-41e2-90bb-9600e443f7b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:26:34.847602 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.847586    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e2bb450-f2a0-41e2-90bb-9600e443f7b2-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:34.849258 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.849233    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2bb450-f2a0-41e2-90bb-9600e443f7b2-kube-api-access-2znzk" pod "5e2bb450-f2a0-41e2-90bb-9600e443f7b2" (UID: "5e2bb450-f2a0-41e2-90bb-9600e443f7b2"). InnerVolumeSpecName "kube-api-access-2znzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:26:34.849258 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.849250    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e2bb450-f2a0-41e2-90bb-9600e443f7b2-maas-api-xks-ca" pod "5e2bb450-f2a0-41e2-90bb-9600e443f7b2" (UID: "5e2bb450-f2a0-41e2-90bb-9600e443f7b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:26:34.927383 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.927361    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds"]
Nov 03 08:26:34.927971 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:26:34.927941    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d88475_e8f2_42dc_b2c9_420e69c64636.slice/crio-e7a0f4d9b160d93ed509c5834cf4a0ada974a59abfc9c8325be10254adf66ae9 WatchSource:0}: Error finding container e7a0f4d9b160d93ed509c5834cf4a0ada974a59abfc9c8325be10254adf66ae9: Status 404 returned error can't find the container with id e7a0f4d9b160d93ed509c5834cf4a0ada974a59abfc9c8325be10254adf66ae9
Nov 03 08:26:34.949897 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.949874    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e2bb450-f2a0-41e2-90bb-9600e443f7b2-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:34.950018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:34.949906    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2znzk\" (UniqueName: \"kubernetes.io/projected/5e2bb450-f2a0-41e2-90bb-9600e443f7b2-kube-api-access-2znzk\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:35.605121 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:35.605084    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds" event={"ID":"18d88475-e8f2-42dc-b2c9-420e69c64636","Type":"ContainerStarted","Data":"a83b915f0fc98c2060e6d64c080f44c9ab1ade1e1a510c4943b1870ed455af90"}
Nov 03 08:26:35.605296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:35.605152    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds" event={"ID":"18d88475-e8f2-42dc-b2c9-420e69c64636","Type":"ContainerStarted","Data":"e7a0f4d9b160d93ed509c5834cf4a0ada974a59abfc9c8325be10254adf66ae9"}
Nov 03 08:26:35.606660 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:35.606635    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh" event={"ID":"5e2bb450-f2a0-41e2-90bb-9600e443f7b2","Type":"ContainerDied","Data":"f5faa3415138c5b32fac33a76223968f9c34d3b54a1f0365de7b1ec2ab918ca1"}
Nov 03 08:26:35.606660 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:35.606651    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh"
Nov 03 08:26:35.606819 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:35.606680    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5faa3415138c5b32fac33a76223968f9c34d3b54a1f0365de7b1ec2ab918ca1"
Nov 03 08:26:35.623377 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:35.623327    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds" podStartSLOduration=1.623314656 podStartE2EDuration="1.623314656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:26:35.620285303 +0000 UTC m=+1993.734749234" watchObservedRunningTime="2026-11-03 08:26:35.623314656 +0000 UTC m=+1993.737778586"
Nov 03 08:26:36.616095 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:36.616062    5070 generic.go:356] "Generic (PLEG): container finished" podID="18d88475-e8f2-42dc-b2c9-420e69c64636" containerID="a83b915f0fc98c2060e6d64c080f44c9ab1ade1e1a510c4943b1870ed455af90" exitCode=0
Nov 03 08:26:36.616461 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:36.616114    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds" event={"ID":"18d88475-e8f2-42dc-b2c9-420e69c64636","Type":"ContainerDied","Data":"a83b915f0fc98c2060e6d64c080f44c9ab1ade1e1a510c4943b1870ed455af90"}
Nov 03 08:26:37.024031 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:37.023957    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e620a55-vb8sh"]
Nov 03 08:26:37.027265 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:37.027242    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e620a55-vb8sh"]
Nov 03 08:26:37.789999 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:37.789973    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds"
Nov 03 08:26:37.888411 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:37.888335    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d88475-e8f2-42dc-b2c9-420e69c64636-kube-api-access-cqts6\" (UniqueName: \"kubernetes.io/projected/18d88475-e8f2-42dc-b2c9-420e69c64636-kube-api-access-cqts6\") pod \"18d88475-e8f2-42dc-b2c9-420e69c64636\" (UID: \"18d88475-e8f2-42dc-b2c9-420e69c64636\") "
Nov 03 08:26:37.888411 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:37.888405    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18d88475-e8f2-42dc-b2c9-420e69c64636-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d88475-e8f2-42dc-b2c9-420e69c64636-maas-api-xks-ca\") pod \"18d88475-e8f2-42dc-b2c9-420e69c64636\" (UID: \"18d88475-e8f2-42dc-b2c9-420e69c64636\") "
Nov 03 08:26:37.888629 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:37.888451    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18d88475-e8f2-42dc-b2c9-420e69c64636-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d88475-e8f2-42dc-b2c9-420e69c64636-maas-api-openshift-ca\") pod \"18d88475-e8f2-42dc-b2c9-420e69c64636\" (UID: \"18d88475-e8f2-42dc-b2c9-420e69c64636\") "
Nov 03 08:26:37.888853 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:37.888828    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18d88475-e8f2-42dc-b2c9-420e69c64636-maas-api-openshift-ca" pod "18d88475-e8f2-42dc-b2c9-420e69c64636" (UID: "18d88475-e8f2-42dc-b2c9-420e69c64636"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:26:37.888941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:37.888928    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d88475-e8f2-42dc-b2c9-420e69c64636-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:37.890480 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:37.890456    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18d88475-e8f2-42dc-b2c9-420e69c64636-maas-api-xks-ca" pod "18d88475-e8f2-42dc-b2c9-420e69c64636" (UID: "18d88475-e8f2-42dc-b2c9-420e69c64636"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:26:37.890591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:37.890480    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d88475-e8f2-42dc-b2c9-420e69c64636-kube-api-access-cqts6" pod "18d88475-e8f2-42dc-b2c9-420e69c64636" (UID: "18d88475-e8f2-42dc-b2c9-420e69c64636"). InnerVolumeSpecName "kube-api-access-cqts6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:26:37.990065 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:37.990029    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqts6\" (UniqueName: \"kubernetes.io/projected/18d88475-e8f2-42dc-b2c9-420e69c64636-kube-api-access-cqts6\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:37.990065 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:37.990062    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d88475-e8f2-42dc-b2c9-420e69c64636-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:38.025993 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:38.025965    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7" path="/var/lib/kubelet/pods/401e341b-69d7-4c3f-ad3c-9dcc5b1dc8b7/volumes"
Nov 03 08:26:38.632452 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:38.632425    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds"
Nov 03 08:26:38.632655 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:38.632422    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds" event={"ID":"18d88475-e8f2-42dc-b2c9-420e69c64636","Type":"ContainerDied","Data":"e7a0f4d9b160d93ed509c5834cf4a0ada974a59abfc9c8325be10254adf66ae9"}
Nov 03 08:26:38.632655 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:38.632579    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7a0f4d9b160d93ed509c5834cf4a0ada974a59abfc9c8325be10254adf66ae9"
Nov 03 08:26:48.693036 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.693003    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg"]
Nov 03 08:26:48.694696 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.694394    5070 cpu_manager.go:414] "containerMap: removing container" podUID="5e2bb450-f2a0-41e2-90bb-9600e443f7b2" containerName="revoke-keys"
Nov 03 08:26:48.694696 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.694420    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e2bb450-f2a0-41e2-90bb-9600e443f7b2" containerName="revoke-keys"
Nov 03 08:26:48.694696 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.694439    5070 cpu_manager.go:414] "containerMap: removing container" podUID="18d88475-e8f2-42dc-b2c9-420e69c64636" containerName="revoke-keys"
Nov 03 08:26:48.694696 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.694447    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18d88475-e8f2-42dc-b2c9-420e69c64636" containerName="revoke-keys"
Nov 03 08:26:48.695042 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.694853    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e2bb450-f2a0-41e2-90bb-9600e443f7b2" containerName="revoke-keys"
Nov 03 08:26:48.695042 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.694874    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="18d88475-e8f2-42dc-b2c9-420e69c64636" containerName="revoke-keys"
Nov 03 08:26:48.699170 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.699145    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg"
Nov 03 08:26:48.704446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.704413    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:26:48.704592 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.704476    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:26:48.709005 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.708974    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg"]
Nov 03 08:26:48.719799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.718828    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:26:48.719799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.718945    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:26:48.819843 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.819818    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/510e7368-f5d8-4c33-b226-5c996f9b81c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b02faba-j8hkg\" (UID: \"510e7368-f5d8-4c33-b226-5c996f9b81c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg"
Nov 03 08:26:48.820003 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.819983    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/510e7368-f5d8-4c33-b226-5c996f9b81c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b02faba-j8hkg\" (UID: \"510e7368-f5d8-4c33-b226-5c996f9b81c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg"
Nov 03 08:26:48.820068 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.820028    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpl6x\" (UniqueName: \"kubernetes.io/projected/510e7368-f5d8-4c33-b226-5c996f9b81c0-kube-api-access-fpl6x\") pod \"maas-api-revoke-sub-3b02faba-j8hkg\" (UID: \"510e7368-f5d8-4c33-b226-5c996f9b81c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg"
Nov 03 08:26:48.921103 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.921074    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/510e7368-f5d8-4c33-b226-5c996f9b81c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b02faba-j8hkg\" (UID: \"510e7368-f5d8-4c33-b226-5c996f9b81c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg"
Nov 03 08:26:48.921239 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.921124    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpl6x\" (UniqueName: \"kubernetes.io/projected/510e7368-f5d8-4c33-b226-5c996f9b81c0-kube-api-access-fpl6x\") pod \"maas-api-revoke-sub-3b02faba-j8hkg\" (UID: \"510e7368-f5d8-4c33-b226-5c996f9b81c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg"
Nov 03 08:26:48.921239 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.921184    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/510e7368-f5d8-4c33-b226-5c996f9b81c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b02faba-j8hkg\" (UID: \"510e7368-f5d8-4c33-b226-5c996f9b81c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg"
Nov 03 08:26:48.921912 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.921890    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/510e7368-f5d8-4c33-b226-5c996f9b81c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b02faba-j8hkg\" (UID: \"510e7368-f5d8-4c33-b226-5c996f9b81c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg"
Nov 03 08:26:48.923556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.923530    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/510e7368-f5d8-4c33-b226-5c996f9b81c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b02faba-j8hkg\" (UID: \"510e7368-f5d8-4c33-b226-5c996f9b81c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg"
Nov 03 08:26:48.928508 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:48.928485    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpl6x\" (UniqueName: \"kubernetes.io/projected/510e7368-f5d8-4c33-b226-5c996f9b81c0-kube-api-access-fpl6x\") pod \"maas-api-revoke-sub-3b02faba-j8hkg\" (UID: \"510e7368-f5d8-4c33-b226-5c996f9b81c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg"
Nov 03 08:26:49.022219 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:49.022139    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg"
Nov 03 08:26:49.259484 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:49.259450    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf"]
Nov 03 08:26:49.263941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:49.263923    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf"
Nov 03 08:26:49.275998 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:49.275433    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf"]
Nov 03 08:26:49.329356 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:49.329327    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg"]
Nov 03 08:26:49.329843 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:26:49.329815    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510e7368_f5d8_4c33_b226_5c996f9b81c0.slice/crio-e12e4e6db1e9d1c0621b9eb27704649d71f117db4ce2d703763d32084b3fae46 WatchSource:0}: Error finding container e12e4e6db1e9d1c0621b9eb27704649d71f117db4ce2d703763d32084b3fae46: Status 404 returned error can't find the container with id e12e4e6db1e9d1c0621b9eb27704649d71f117db4ce2d703763d32084b3fae46
Nov 03 08:26:49.453385 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:49.453350    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qxvf\" (UniqueName: \"kubernetes.io/projected/7b0903f1-a417-4c98-932e-0a2ac9115d63-kube-api-access-6qxvf\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf\" (UID: \"7b0903f1-a417-4c98-932e-0a2ac9115d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf"
Nov 03 08:26:49.453589 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:49.453529    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0903f1-a417-4c98-932e-0a2ac9115d63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf\" (UID: \"7b0903f1-a417-4c98-932e-0a2ac9115d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf"
Nov 03 08:26:49.453689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:49.453627    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0903f1-a417-4c98-932e-0a2ac9115d63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf\" (UID: \"7b0903f1-a417-4c98-932e-0a2ac9115d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf"
Nov 03 08:26:49.555351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:49.555276    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qxvf\" (UniqueName: \"kubernetes.io/projected/7b0903f1-a417-4c98-932e-0a2ac9115d63-kube-api-access-6qxvf\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf\" (UID: \"7b0903f1-a417-4c98-932e-0a2ac9115d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf"
Nov 03 08:26:49.555487 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:49.555362    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0903f1-a417-4c98-932e-0a2ac9115d63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf\" (UID: \"7b0903f1-a417-4c98-932e-0a2ac9115d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf"
Nov 03 08:26:49.555487 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:49.555408    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0903f1-a417-4c98-932e-0a2ac9115d63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf\" (UID: \"7b0903f1-a417-4c98-932e-0a2ac9115d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf"
Nov 03 08:26:49.556045 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:49.556020    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0903f1-a417-4c98-932e-0a2ac9115d63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf\" (UID: \"7b0903f1-a417-4c98-932e-0a2ac9115d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf"
Nov 03 08:26:49.557907 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:49.557881    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0903f1-a417-4c98-932e-0a2ac9115d63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf\" (UID: \"7b0903f1-a417-4c98-932e-0a2ac9115d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf"
Nov 03 08:26:49.563915 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:49.563892    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qxvf\" (UniqueName: \"kubernetes.io/projected/7b0903f1-a417-4c98-932e-0a2ac9115d63-kube-api-access-6qxvf\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf\" (UID: \"7b0903f1-a417-4c98-932e-0a2ac9115d63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf"
Nov 03 08:26:49.580562 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:49.580541    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf"
Nov 03 08:26:49.728753 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:49.728708    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg" event={"ID":"510e7368-f5d8-4c33-b226-5c996f9b81c0","Type":"ContainerStarted","Data":"a2cafe5cdc571bcb38765a29cb262319d935b9223c6d13313952f7dbb8efac3a"}
Nov 03 08:26:49.729689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:49.728795    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg" event={"ID":"510e7368-f5d8-4c33-b226-5c996f9b81c0","Type":"ContainerStarted","Data":"e12e4e6db1e9d1c0621b9eb27704649d71f117db4ce2d703763d32084b3fae46"}
Nov 03 08:26:49.770385 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:49.770342    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg" podStartSLOduration=1.77032905 podStartE2EDuration="1.77032905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:26:49.766248119 +0000 UTC m=+2007.880712055" watchObservedRunningTime="2026-11-03 08:26:49.77032905 +0000 UTC m=+2007.884793020"
Nov 03 08:26:49.868969 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:26:49.868939    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0903f1_a417_4c98_932e_0a2ac9115d63.slice/crio-3def1c6ba2b696e9dbcd5adcbbb04e0694e9637c0f5b73ce2c25db77cfa667cb WatchSource:0}: Error finding container 3def1c6ba2b696e9dbcd5adcbbb04e0694e9637c0f5b73ce2c25db77cfa667cb: Status 404 returned error can't find the container with id 3def1c6ba2b696e9dbcd5adcbbb04e0694e9637c0f5b73ce2c25db77cfa667cb
Nov 03 08:26:49.869575 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:49.869553    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf"]
Nov 03 08:26:50.737503 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:50.737471    5070 generic.go:356] "Generic (PLEG): container finished" podID="510e7368-f5d8-4c33-b226-5c996f9b81c0" containerID="a2cafe5cdc571bcb38765a29cb262319d935b9223c6d13313952f7dbb8efac3a" exitCode=0
Nov 03 08:26:50.737943 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:50.737518    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg" event={"ID":"510e7368-f5d8-4c33-b226-5c996f9b81c0","Type":"ContainerDied","Data":"a2cafe5cdc571bcb38765a29cb262319d935b9223c6d13313952f7dbb8efac3a"}
Nov 03 08:26:50.739475 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:50.739453    5070 generic.go:356] "Generic (PLEG): container finished" podID="7b0903f1-a417-4c98-932e-0a2ac9115d63" containerID="d18f506bb2de0a8e812b8e168c256bf8aadb09fdca38cb22b50056dfdd62dd27" exitCode=0
Nov 03 08:26:50.739596 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:50.739515    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf" event={"ID":"7b0903f1-a417-4c98-932e-0a2ac9115d63","Type":"ContainerDied","Data":"d18f506bb2de0a8e812b8e168c256bf8aadb09fdca38cb22b50056dfdd62dd27"}
Nov 03 08:26:50.739596 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:50.739561    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf" event={"ID":"7b0903f1-a417-4c98-932e-0a2ac9115d63","Type":"ContainerStarted","Data":"3def1c6ba2b696e9dbcd5adcbbb04e0694e9637c0f5b73ce2c25db77cfa667cb"}
Nov 03 08:26:51.938039 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:51.938014    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg"
Nov 03 08:26:51.941643 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:51.941622    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf"
Nov 03 08:26:52.052037 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:26:52.051964    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:26:52.086821 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.086721    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b0903f1-a417-4c98-932e-0a2ac9115d63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0903f1-a417-4c98-932e-0a2ac9115d63-maas-api-xks-ca\") pod \"7b0903f1-a417-4c98-932e-0a2ac9115d63\" (UID: \"7b0903f1-a417-4c98-932e-0a2ac9115d63\") "
Nov 03 08:26:52.086821 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.086803    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510e7368-f5d8-4c33-b226-5c996f9b81c0-kube-api-access-fpl6x\" (UniqueName: \"kubernetes.io/projected/510e7368-f5d8-4c33-b226-5c996f9b81c0-kube-api-access-fpl6x\") pod \"510e7368-f5d8-4c33-b226-5c996f9b81c0\" (UID: \"510e7368-f5d8-4c33-b226-5c996f9b81c0\") "
Nov 03 08:26:52.087156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.086853    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b0903f1-a417-4c98-932e-0a2ac9115d63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0903f1-a417-4c98-932e-0a2ac9115d63-maas-api-openshift-ca\") pod \"7b0903f1-a417-4c98-932e-0a2ac9115d63\" (UID: \"7b0903f1-a417-4c98-932e-0a2ac9115d63\") "
Nov 03 08:26:52.087156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.086891    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/510e7368-f5d8-4c33-b226-5c996f9b81c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/510e7368-f5d8-4c33-b226-5c996f9b81c0-maas-api-xks-ca\") pod \"510e7368-f5d8-4c33-b226-5c996f9b81c0\" (UID: \"510e7368-f5d8-4c33-b226-5c996f9b81c0\") "
Nov 03 08:26:52.087156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.086923    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/510e7368-f5d8-4c33-b226-5c996f9b81c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/510e7368-f5d8-4c33-b226-5c996f9b81c0-maas-api-openshift-ca\") pod \"510e7368-f5d8-4c33-b226-5c996f9b81c0\" (UID: \"510e7368-f5d8-4c33-b226-5c996f9b81c0\") "
Nov 03 08:26:52.087156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.086958    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b0903f1-a417-4c98-932e-0a2ac9115d63-kube-api-access-6qxvf\" (UniqueName: \"kubernetes.io/projected/7b0903f1-a417-4c98-932e-0a2ac9115d63-kube-api-access-6qxvf\") pod \"7b0903f1-a417-4c98-932e-0a2ac9115d63\" (UID: \"7b0903f1-a417-4c98-932e-0a2ac9115d63\") "
Nov 03 08:26:52.087466 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.087254    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0903f1-a417-4c98-932e-0a2ac9115d63-maas-api-openshift-ca" pod "7b0903f1-a417-4c98-932e-0a2ac9115d63" (UID: "7b0903f1-a417-4c98-932e-0a2ac9115d63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:26:52.087466 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.087309    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/510e7368-f5d8-4c33-b226-5c996f9b81c0-maas-api-openshift-ca" pod "510e7368-f5d8-4c33-b226-5c996f9b81c0" (UID: "510e7368-f5d8-4c33-b226-5c996f9b81c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:26:52.087580 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.087473    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0903f1-a417-4c98-932e-0a2ac9115d63-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:52.087580 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.087497    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/510e7368-f5d8-4c33-b226-5c996f9b81c0-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:52.089278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.089243    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/510e7368-f5d8-4c33-b226-5c996f9b81c0-maas-api-xks-ca" pod "510e7368-f5d8-4c33-b226-5c996f9b81c0" (UID: "510e7368-f5d8-4c33-b226-5c996f9b81c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:26:52.089461 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.089440    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0903f1-a417-4c98-932e-0a2ac9115d63-kube-api-access-6qxvf" pod "7b0903f1-a417-4c98-932e-0a2ac9115d63" (UID: "7b0903f1-a417-4c98-932e-0a2ac9115d63"). InnerVolumeSpecName "kube-api-access-6qxvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:26:52.089556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.089535    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b0903f1-a417-4c98-932e-0a2ac9115d63-maas-api-xks-ca" pod "7b0903f1-a417-4c98-932e-0a2ac9115d63" (UID: "7b0903f1-a417-4c98-932e-0a2ac9115d63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:26:52.089600 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.089555    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510e7368-f5d8-4c33-b226-5c996f9b81c0-kube-api-access-fpl6x" pod "510e7368-f5d8-4c33-b226-5c996f9b81c0" (UID: "510e7368-f5d8-4c33-b226-5c996f9b81c0"). InnerVolumeSpecName "kube-api-access-fpl6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:26:52.188846 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.188820    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0903f1-a417-4c98-932e-0a2ac9115d63-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:52.188846 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.188844    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpl6x\" (UniqueName: \"kubernetes.io/projected/510e7368-f5d8-4c33-b226-5c996f9b81c0-kube-api-access-fpl6x\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:52.189063 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.188855    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/510e7368-f5d8-4c33-b226-5c996f9b81c0-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:52.189063 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.188863    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qxvf\" (UniqueName: \"kubernetes.io/projected/7b0903f1-a417-4c98-932e-0a2ac9115d63-kube-api-access-6qxvf\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:26:52.762510 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.762479    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg"
Nov 03 08:26:52.762695 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.762505    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg" event={"ID":"510e7368-f5d8-4c33-b226-5c996f9b81c0","Type":"ContainerDied","Data":"e12e4e6db1e9d1c0621b9eb27704649d71f117db4ce2d703763d32084b3fae46"}
Nov 03 08:26:52.762695 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.762556    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e12e4e6db1e9d1c0621b9eb27704649d71f117db4ce2d703763d32084b3fae46"
Nov 03 08:26:52.764121 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.764102    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf"
Nov 03 08:26:52.764121 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.764108    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf" event={"ID":"7b0903f1-a417-4c98-932e-0a2ac9115d63","Type":"ContainerDied","Data":"3def1c6ba2b696e9dbcd5adcbbb04e0694e9637c0f5b73ce2c25db77cfa667cb"}
Nov 03 08:26:52.764296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:52.764153    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3def1c6ba2b696e9dbcd5adcbbb04e0694e9637c0f5b73ce2c25db77cfa667cb"
Nov 03 08:26:55.990430 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:55.990375    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:26:59.629492 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.629454    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg"]
Nov 03 08:26:59.630281 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.630259    5070 cpu_manager.go:414] "containerMap: removing container" podUID="510e7368-f5d8-4c33-b226-5c996f9b81c0" containerName="revoke-keys"
Nov 03 08:26:59.630392 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.630282    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510e7368-f5d8-4c33-b226-5c996f9b81c0" containerName="revoke-keys"
Nov 03 08:26:59.630392 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.630306    5070 cpu_manager.go:414] "containerMap: removing container" podUID="7b0903f1-a417-4c98-932e-0a2ac9115d63" containerName="revoke-keys"
Nov 03 08:26:59.630392 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.630314    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b0903f1-a417-4c98-932e-0a2ac9115d63" containerName="revoke-keys"
Nov 03 08:26:59.630563 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.630483    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="510e7368-f5d8-4c33-b226-5c996f9b81c0" containerName="revoke-keys"
Nov 03 08:26:59.630563 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.630501    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b0903f1-a417-4c98-932e-0a2ac9115d63" containerName="revoke-keys"
Nov 03 08:26:59.635576 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.635548    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg"
Nov 03 08:26:59.643932 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.642170    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:26:59.643932 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.642312    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:26:59.643932 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.642403    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:26:59.643932 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.642460    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:26:59.646689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.646659    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg"]
Nov 03 08:26:59.679591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.679553    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffa08f24-2c32-4d6b-89b5-61ec07971549-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg\" (UID: \"ffa08f24-2c32-4d6b-89b5-61ec07971549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg"
Nov 03 08:26:59.679793 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.679629    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffa08f24-2c32-4d6b-89b5-61ec07971549-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg\" (UID: \"ffa08f24-2c32-4d6b-89b5-61ec07971549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg"
Nov 03 08:26:59.679793 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.679687    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxccz\" (UniqueName: \"kubernetes.io/projected/ffa08f24-2c32-4d6b-89b5-61ec07971549-kube-api-access-sxccz\") pod \"maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg\" (UID: \"ffa08f24-2c32-4d6b-89b5-61ec07971549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg"
Nov 03 08:26:59.782299 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.782256    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffa08f24-2c32-4d6b-89b5-61ec07971549-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg\" (UID: \"ffa08f24-2c32-4d6b-89b5-61ec07971549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg"
Nov 03 08:26:59.782489 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.782342    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxccz\" (UniqueName: \"kubernetes.io/projected/ffa08f24-2c32-4d6b-89b5-61ec07971549-kube-api-access-sxccz\") pod \"maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg\" (UID: \"ffa08f24-2c32-4d6b-89b5-61ec07971549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg"
Nov 03 08:26:59.782581 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.782565    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffa08f24-2c32-4d6b-89b5-61ec07971549-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg\" (UID: \"ffa08f24-2c32-4d6b-89b5-61ec07971549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg"
Nov 03 08:26:59.783407 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.783382    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffa08f24-2c32-4d6b-89b5-61ec07971549-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg\" (UID: \"ffa08f24-2c32-4d6b-89b5-61ec07971549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg"
Nov 03 08:26:59.791824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.791719    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxccz\" (UniqueName: \"kubernetes.io/projected/ffa08f24-2c32-4d6b-89b5-61ec07971549-kube-api-access-sxccz\") pod \"maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg\" (UID: \"ffa08f24-2c32-4d6b-89b5-61ec07971549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg"
Nov 03 08:26:59.797417 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.797388    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffa08f24-2c32-4d6b-89b5-61ec07971549-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg\" (UID: \"ffa08f24-2c32-4d6b-89b5-61ec07971549\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg"
Nov 03 08:26:59.964857 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:26:59.964784    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg"
Nov 03 08:27:00.260472 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:00.260442    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg"]
Nov 03 08:27:00.261537 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:27:00.261515    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa08f24_2c32_4d6b_89b5_61ec07971549.slice/crio-f7376d438185d3ebcd52b83238227f17267f25dc574ba9ecb7d7b5dc68762115 WatchSource:0}: Error finding container f7376d438185d3ebcd52b83238227f17267f25dc574ba9ecb7d7b5dc68762115: Status 404 returned error can't find the container with id f7376d438185d3ebcd52b83238227f17267f25dc574ba9ecb7d7b5dc68762115
Nov 03 08:27:00.830824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:00.830720    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg" event={"ID":"ffa08f24-2c32-4d6b-89b5-61ec07971549","Type":"ContainerStarted","Data":"0619387f1aeafe6242c20e572bb78364964948700157b66b8585af08a38b8723"}
Nov 03 08:27:00.830824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:00.830800    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg" event={"ID":"ffa08f24-2c32-4d6b-89b5-61ec07971549","Type":"ContainerStarted","Data":"f7376d438185d3ebcd52b83238227f17267f25dc574ba9ecb7d7b5dc68762115"}
Nov 03 08:27:00.850634 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:00.850589    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg" podStartSLOduration=1.85057656 podStartE2EDuration="1.85057656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:27:00.848129556 +0000 UTC m=+2018.962593486" watchObservedRunningTime="2026-11-03 08:27:00.85057656 +0000 UTC m=+2018.965040491"
Nov 03 08:27:01.842325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:01.842288    5070 generic.go:356] "Generic (PLEG): container finished" podID="ffa08f24-2c32-4d6b-89b5-61ec07971549" containerID="0619387f1aeafe6242c20e572bb78364964948700157b66b8585af08a38b8723" exitCode=0
Nov 03 08:27:01.843111 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:01.842373    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg" event={"ID":"ffa08f24-2c32-4d6b-89b5-61ec07971549","Type":"ContainerDied","Data":"0619387f1aeafe6242c20e572bb78364964948700157b66b8585af08a38b8723"}
Nov 03 08:27:01.900356 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:01.900314    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf"]
Nov 03 08:27:01.904578 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:01.904552    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-a61a48-4plqf"]
Nov 03 08:27:02.028888 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:02.028857    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b0903f1-a417-4c98-932e-0a2ac9115d63" path="/var/lib/kubelet/pods/7b0903f1-a417-4c98-932e-0a2ac9115d63/volumes"
Nov 03 08:27:03.023565 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:03.023542    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg"
Nov 03 08:27:03.024658 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:03.024639    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffa08f24-2c32-4d6b-89b5-61ec07971549-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffa08f24-2c32-4d6b-89b5-61ec07971549-maas-api-xks-ca\") pod \"ffa08f24-2c32-4d6b-89b5-61ec07971549\" (UID: \"ffa08f24-2c32-4d6b-89b5-61ec07971549\") "
Nov 03 08:27:03.024728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:03.024694    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffa08f24-2c32-4d6b-89b5-61ec07971549-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffa08f24-2c32-4d6b-89b5-61ec07971549-maas-api-openshift-ca\") pod \"ffa08f24-2c32-4d6b-89b5-61ec07971549\" (UID: \"ffa08f24-2c32-4d6b-89b5-61ec07971549\") "
Nov 03 08:27:03.024807 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:03.024776    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffa08f24-2c32-4d6b-89b5-61ec07971549-kube-api-access-sxccz\" (UniqueName: \"kubernetes.io/projected/ffa08f24-2c32-4d6b-89b5-61ec07971549-kube-api-access-sxccz\") pod \"ffa08f24-2c32-4d6b-89b5-61ec07971549\" (UID: \"ffa08f24-2c32-4d6b-89b5-61ec07971549\") "
Nov 03 08:27:03.025141 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:03.025118    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa08f24-2c32-4d6b-89b5-61ec07971549-maas-api-openshift-ca" pod "ffa08f24-2c32-4d6b-89b5-61ec07971549" (UID: "ffa08f24-2c32-4d6b-89b5-61ec07971549"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:27:03.025416 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:03.025375    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffa08f24-2c32-4d6b-89b5-61ec07971549-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:03.026957 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:03.026929    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa08f24-2c32-4d6b-89b5-61ec07971549-maas-api-xks-ca" pod "ffa08f24-2c32-4d6b-89b5-61ec07971549" (UID: "ffa08f24-2c32-4d6b-89b5-61ec07971549"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:03.027066 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:03.027031    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa08f24-2c32-4d6b-89b5-61ec07971549-kube-api-access-sxccz" pod "ffa08f24-2c32-4d6b-89b5-61ec07971549" (UID: "ffa08f24-2c32-4d6b-89b5-61ec07971549"). InnerVolumeSpecName "kube-api-access-sxccz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:03.126476 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:03.126416    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxccz\" (UniqueName: \"kubernetes.io/projected/ffa08f24-2c32-4d6b-89b5-61ec07971549-kube-api-access-sxccz\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:03.126476 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:03.126442    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffa08f24-2c32-4d6b-89b5-61ec07971549-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:03.859020 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:03.858992    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg"
Nov 03 08:27:03.859237 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:03.859029    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg" event={"ID":"ffa08f24-2c32-4d6b-89b5-61ec07971549","Type":"ContainerDied","Data":"f7376d438185d3ebcd52b83238227f17267f25dc574ba9ecb7d7b5dc68762115"}
Nov 03 08:27:03.859237 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:03.859082    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7376d438185d3ebcd52b83238227f17267f25dc574ba9ecb7d7b5dc68762115"
Nov 03 08:27:09.868016 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:09.867983    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq"]
Nov 03 08:27:09.881872 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:09.881838    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s"]
Nov 03 08:27:10.026188 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.026157    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd"]
Nov 03 08:27:10.026672 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.026659    5070 cpu_manager.go:414] "containerMap: removing container" podUID="ffa08f24-2c32-4d6b-89b5-61ec07971549" containerName="revoke-keys"
Nov 03 08:27:10.026729 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.026674    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffa08f24-2c32-4d6b-89b5-61ec07971549" containerName="revoke-keys"
Nov 03 08:27:10.026809 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.026799    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffa08f24-2c32-4d6b-89b5-61ec07971549" containerName="revoke-keys"
Nov 03 08:27:10.030289 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.030268    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd"
Nov 03 08:27:10.035016 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.034992    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd"]
Nov 03 08:27:10.038810 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.038480    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:27:10.038810 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.038632    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:27:10.038810 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.038659    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:27:10.038810 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.038721    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:27:10.108399 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.108370    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kfjh\" (UniqueName: \"kubernetes.io/projected/4a176d4a-4a2a-425e-b6e6-bd21eaed8669-kube-api-access-4kfjh\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd\" (UID: \"4a176d4a-4a2a-425e-b6e6-bd21eaed8669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd"
Nov 03 08:27:10.108549 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.108463    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a176d4a-4a2a-425e-b6e6-bd21eaed8669-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd\" (UID: \"4a176d4a-4a2a-425e-b6e6-bd21eaed8669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd"
Nov 03 08:27:10.108635 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.108614    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a176d4a-4a2a-425e-b6e6-bd21eaed8669-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd\" (UID: \"4a176d4a-4a2a-425e-b6e6-bd21eaed8669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd"
Nov 03 08:27:10.210638 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.210533    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kfjh\" (UniqueName: \"kubernetes.io/projected/4a176d4a-4a2a-425e-b6e6-bd21eaed8669-kube-api-access-4kfjh\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd\" (UID: \"4a176d4a-4a2a-425e-b6e6-bd21eaed8669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd"
Nov 03 08:27:10.210852 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.210653    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a176d4a-4a2a-425e-b6e6-bd21eaed8669-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd\" (UID: \"4a176d4a-4a2a-425e-b6e6-bd21eaed8669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd"
Nov 03 08:27:10.210852 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.210821    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a176d4a-4a2a-425e-b6e6-bd21eaed8669-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd\" (UID: \"4a176d4a-4a2a-425e-b6e6-bd21eaed8669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd"
Nov 03 08:27:10.211533 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.211502    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a176d4a-4a2a-425e-b6e6-bd21eaed8669-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd\" (UID: \"4a176d4a-4a2a-425e-b6e6-bd21eaed8669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd"
Nov 03 08:27:10.214325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.214297    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a176d4a-4a2a-425e-b6e6-bd21eaed8669-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd\" (UID: \"4a176d4a-4a2a-425e-b6e6-bd21eaed8669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd"
Nov 03 08:27:10.219583 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.219556    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kfjh\" (UniqueName: \"kubernetes.io/projected/4a176d4a-4a2a-425e-b6e6-bd21eaed8669-kube-api-access-4kfjh\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd\" (UID: \"4a176d4a-4a2a-425e-b6e6-bd21eaed8669\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd"
Nov 03 08:27:10.347682 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.347654    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd"
Nov 03 08:27:10.628929 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.628892    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd"]
Nov 03 08:27:10.629998 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:27:10.629961    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a176d4a_4a2a_425e_b6e6_bd21eaed8669.slice/crio-82cf68883ce52a18733f397c823a30cf33daeab34eed4d90964678c58b770292 WatchSource:0}: Error finding container 82cf68883ce52a18733f397c823a30cf33daeab34eed4d90964678c58b770292: Status 404 returned error can't find the container with id 82cf68883ce52a18733f397c823a30cf33daeab34eed4d90964678c58b770292
Nov 03 08:27:10.921291 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:10.921217    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd" event={"ID":"4a176d4a-4a2a-425e-b6e6-bd21eaed8669","Type":"ContainerStarted","Data":"82cf68883ce52a18733f397c823a30cf33daeab34eed4d90964678c58b770292"}
Nov 03 08:27:11.930144 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:11.930110    5070 generic.go:356] "Generic (PLEG): container finished" podID="4a176d4a-4a2a-425e-b6e6-bd21eaed8669" containerID="3c2016c13fd48186b9af89497995b7770d5789016ee1cb92f5a22ce6e80f06df" exitCode=0
Nov 03 08:27:11.930527 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:11.930186    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd" event={"ID":"4a176d4a-4a2a-425e-b6e6-bd21eaed8669","Type":"ContainerDied","Data":"3c2016c13fd48186b9af89497995b7770d5789016ee1cb92f5a22ce6e80f06df"}
Nov 03 08:27:13.101566 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:13.101539    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd"
Nov 03 08:27:13.146663 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:13.146635    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a176d4a-4a2a-425e-b6e6-bd21eaed8669-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a176d4a-4a2a-425e-b6e6-bd21eaed8669-maas-api-xks-ca\") pod \"4a176d4a-4a2a-425e-b6e6-bd21eaed8669\" (UID: \"4a176d4a-4a2a-425e-b6e6-bd21eaed8669\") "
Nov 03 08:27:13.146837 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:13.146706    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a176d4a-4a2a-425e-b6e6-bd21eaed8669-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a176d4a-4a2a-425e-b6e6-bd21eaed8669-maas-api-openshift-ca\") pod \"4a176d4a-4a2a-425e-b6e6-bd21eaed8669\" (UID: \"4a176d4a-4a2a-425e-b6e6-bd21eaed8669\") "
Nov 03 08:27:13.146933 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:13.146914    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a176d4a-4a2a-425e-b6e6-bd21eaed8669-kube-api-access-4kfjh\" (UniqueName: \"kubernetes.io/projected/4a176d4a-4a2a-425e-b6e6-bd21eaed8669-kube-api-access-4kfjh\") pod \"4a176d4a-4a2a-425e-b6e6-bd21eaed8669\" (UID: \"4a176d4a-4a2a-425e-b6e6-bd21eaed8669\") "
Nov 03 08:27:13.147140 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:13.147114    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a176d4a-4a2a-425e-b6e6-bd21eaed8669-maas-api-openshift-ca" pod "4a176d4a-4a2a-425e-b6e6-bd21eaed8669" (UID: "4a176d4a-4a2a-425e-b6e6-bd21eaed8669"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:27:13.147904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:13.147883    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a176d4a-4a2a-425e-b6e6-bd21eaed8669-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:13.149044 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:13.149019    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a176d4a-4a2a-425e-b6e6-bd21eaed8669-maas-api-xks-ca" pod "4a176d4a-4a2a-425e-b6e6-bd21eaed8669" (UID: "4a176d4a-4a2a-425e-b6e6-bd21eaed8669"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:13.149667 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:13.149642    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a176d4a-4a2a-425e-b6e6-bd21eaed8669-kube-api-access-4kfjh" pod "4a176d4a-4a2a-425e-b6e6-bd21eaed8669" (UID: "4a176d4a-4a2a-425e-b6e6-bd21eaed8669"). InnerVolumeSpecName "kube-api-access-4kfjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:13.249230 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:13.249170    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a176d4a-4a2a-425e-b6e6-bd21eaed8669-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:13.249230 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:13.249196    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kfjh\" (UniqueName: \"kubernetes.io/projected/4a176d4a-4a2a-425e-b6e6-bd21eaed8669-kube-api-access-4kfjh\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:13.947645 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:13.947616    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd"
Nov 03 08:27:13.947875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:13.947656    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd" event={"ID":"4a176d4a-4a2a-425e-b6e6-bd21eaed8669","Type":"ContainerDied","Data":"82cf68883ce52a18733f397c823a30cf33daeab34eed4d90964678c58b770292"}
Nov 03 08:27:13.947875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:13.947744    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82cf68883ce52a18733f397c823a30cf33daeab34eed4d90964678c58b770292"
Nov 03 08:27:14.433088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:14.433059    5070 scope.go:122] "RemoveContainer" containerID="053a383cb4665a3cbc082e68cfaa70d901567e6b41f8e2a0dbfe49950e1d6d28"
Nov 03 08:27:14.868416 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:14.868370    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq" podUID="6b7840f4-acec-4021-9ab3-c15fc29f5f8b" containerName="payload-processing" containerID="cri-o://79aca1a1021c1bb1cf4fc500127cb274a1d12f6a454e58edf89f65f133fbe158" gracePeriod=25
Nov 03 08:27:14.882894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:14.882857    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s" podUID="84094812-6bd0-401a-a91c-d3d24e71501f" containerName="payload-pre-processing" containerID="cri-o://4df1c0060b45187901e2d6d0a4ef244479c0b5de76163af428e0c148684ba023" gracePeriod=25
Nov 03 08:27:18.126550 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:18.126516    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg"]
Nov 03 08:27:18.131226 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:18.131185    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a66d5c-d4slg"]
Nov 03 08:27:19.534274 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:19.534234    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s" podUID="84094812-6bd0-401a-a91c-d3d24e71501f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:27:19.535639 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:19.535603    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq" podUID="6b7840f4-acec-4021-9ab3-c15fc29f5f8b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:27:20.025614 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:20.025583    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffa08f24-2c32-4d6b-89b5-61ec07971549" path="/var/lib/kubelet/pods/ffa08f24-2c32-4d6b-89b5-61ec07971549/volumes"
Nov 03 08:27:21.078707 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:21.078673    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:27:22.051556 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:27:22.051523    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:27:25.240967 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.240931    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd"]
Nov 03 08:27:25.258807 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.254008    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-d6a8bf-nh9pd"]
Nov 03 08:27:25.442502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.442468    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5"]
Nov 03 08:27:25.443187 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.443165    5070 cpu_manager.go:414] "containerMap: removing container" podUID="4a176d4a-4a2a-425e-b6e6-bd21eaed8669" containerName="revoke-keys"
Nov 03 08:27:25.443187 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.443186    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a176d4a-4a2a-425e-b6e6-bd21eaed8669" containerName="revoke-keys"
Nov 03 08:27:25.443405 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.443375    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a176d4a-4a2a-425e-b6e6-bd21eaed8669" containerName="revoke-keys"
Nov 03 08:27:25.448222 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.448200    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5"
Nov 03 08:27:25.451214 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.451141    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:27:25.451468 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.451440    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:27:25.451624 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.451530    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:27:25.451724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.451684    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:27:25.456683 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.456659    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5"]
Nov 03 08:27:25.623403 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.623366    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e940da6d-ee8d-42f1-bdf2-e580a3705fab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc1552c1-6jwk5\" (UID: \"e940da6d-ee8d-42f1-bdf2-e580a3705fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5"
Nov 03 08:27:25.623576 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.623505    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e940da6d-ee8d-42f1-bdf2-e580a3705fab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc1552c1-6jwk5\" (UID: \"e940da6d-ee8d-42f1-bdf2-e580a3705fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5"
Nov 03 08:27:25.623576 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.623539    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t585r\" (UniqueName: \"kubernetes.io/projected/e940da6d-ee8d-42f1-bdf2-e580a3705fab-kube-api-access-t585r\") pod \"maas-api-revoke-sub-dc1552c1-6jwk5\" (UID: \"e940da6d-ee8d-42f1-bdf2-e580a3705fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5"
Nov 03 08:27:25.724748 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.724700    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e940da6d-ee8d-42f1-bdf2-e580a3705fab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc1552c1-6jwk5\" (UID: \"e940da6d-ee8d-42f1-bdf2-e580a3705fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5"
Nov 03 08:27:25.724925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.724861    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e940da6d-ee8d-42f1-bdf2-e580a3705fab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc1552c1-6jwk5\" (UID: \"e940da6d-ee8d-42f1-bdf2-e580a3705fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5"
Nov 03 08:27:25.724925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.724896    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t585r\" (UniqueName: \"kubernetes.io/projected/e940da6d-ee8d-42f1-bdf2-e580a3705fab-kube-api-access-t585r\") pod \"maas-api-revoke-sub-dc1552c1-6jwk5\" (UID: \"e940da6d-ee8d-42f1-bdf2-e580a3705fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5"
Nov 03 08:27:25.725432 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.725407    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e940da6d-ee8d-42f1-bdf2-e580a3705fab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc1552c1-6jwk5\" (UID: \"e940da6d-ee8d-42f1-bdf2-e580a3705fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5"
Nov 03 08:27:25.727186 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.727165    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e940da6d-ee8d-42f1-bdf2-e580a3705fab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc1552c1-6jwk5\" (UID: \"e940da6d-ee8d-42f1-bdf2-e580a3705fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5"
Nov 03 08:27:25.732275 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.732253    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t585r\" (UniqueName: \"kubernetes.io/projected/e940da6d-ee8d-42f1-bdf2-e580a3705fab-kube-api-access-t585r\") pod \"maas-api-revoke-sub-dc1552c1-6jwk5\" (UID: \"e940da6d-ee8d-42f1-bdf2-e580a3705fab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5"
Nov 03 08:27:25.764863 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:25.764843    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5"
Nov 03 08:27:26.026716 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:26.026641    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a176d4a-4a2a-425e-b6e6-bd21eaed8669" path="/var/lib/kubelet/pods/4a176d4a-4a2a-425e-b6e6-bd21eaed8669/volumes"
Nov 03 08:27:26.081895 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:26.081862    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5"]
Nov 03 08:27:26.082568 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:27:26.082544    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode940da6d_ee8d_42f1_bdf2_e580a3705fab.slice/crio-2c07f2eca3e0db491dfdbdf06775cb2f208fdca8d61df01bcea23b4034362161 WatchSource:0}: Error finding container 2c07f2eca3e0db491dfdbdf06775cb2f208fdca8d61df01bcea23b4034362161: Status 404 returned error can't find the container with id 2c07f2eca3e0db491dfdbdf06775cb2f208fdca8d61df01bcea23b4034362161
Nov 03 08:27:26.513673 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:26.513646    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-109070-ms2xd"]
Nov 03 08:27:26.517717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:26.517700    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-109070-ms2xd"
Nov 03 08:27:26.526482 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:26.526458    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-109070-ms2xd"]
Nov 03 08:27:26.637586 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:26.637503    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983d72df-19d6-4506-9794-ef75dbe012c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-109070-ms2xd\" (UID: \"983d72df-19d6-4506-9794-ef75dbe012c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-109070-ms2xd"
Nov 03 08:27:26.637779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:26.637609    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983d72df-19d6-4506-9794-ef75dbe012c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-109070-ms2xd\" (UID: \"983d72df-19d6-4506-9794-ef75dbe012c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-109070-ms2xd"
Nov 03 08:27:26.637779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:26.637682    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zhv8\" (UniqueName: \"kubernetes.io/projected/983d72df-19d6-4506-9794-ef75dbe012c3-kube-api-access-6zhv8\") pod \"maas-api-revoke-keys-e2e-inf-b-109070-ms2xd\" (UID: \"983d72df-19d6-4506-9794-ef75dbe012c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-109070-ms2xd"
Nov 03 08:27:26.739260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:26.739231    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983d72df-19d6-4506-9794-ef75dbe012c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-109070-ms2xd\" (UID: \"983d72df-19d6-4506-9794-ef75dbe012c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-109070-ms2xd"
Nov 03 08:27:26.739422 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:26.739287    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983d72df-19d6-4506-9794-ef75dbe012c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-109070-ms2xd\" (UID: \"983d72df-19d6-4506-9794-ef75dbe012c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-109070-ms2xd"
Nov 03 08:27:26.739422 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:26.739318    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zhv8\" (UniqueName: \"kubernetes.io/projected/983d72df-19d6-4506-9794-ef75dbe012c3-kube-api-access-6zhv8\") pod \"maas-api-revoke-keys-e2e-inf-b-109070-ms2xd\" (UID: \"983d72df-19d6-4506-9794-ef75dbe012c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-109070-ms2xd"
Nov 03 08:27:26.739994 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:26.739975    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983d72df-19d6-4506-9794-ef75dbe012c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-109070-ms2xd\" (UID: \"983d72df-19d6-4506-9794-ef75dbe012c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-109070-ms2xd"
Nov 03 08:27:26.741602 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:26.741584    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983d72df-19d6-4506-9794-ef75dbe012c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-109070-ms2xd\" (UID: \"983d72df-19d6-4506-9794-ef75dbe012c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-109070-ms2xd"
Nov 03 08:27:26.747230 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:26.747205    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zhv8\" (UniqueName: \"kubernetes.io/projected/983d72df-19d6-4506-9794-ef75dbe012c3-kube-api-access-6zhv8\") pod \"maas-api-revoke-keys-e2e-inf-b-109070-ms2xd\" (UID: \"983d72df-19d6-4506-9794-ef75dbe012c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-109070-ms2xd"
Nov 03 08:27:26.832829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:26.832801    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-109070-ms2xd"
Nov 03 08:27:27.070732 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:27.070689    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5" event={"ID":"e940da6d-ee8d-42f1-bdf2-e580a3705fab","Type":"ContainerStarted","Data":"d6f2c6e93cc28712da8e521fba2ef4cb373740f44f26a4cb9320f6cbf7556ce7"}
Nov 03 08:27:27.070937 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:27.070806    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5" event={"ID":"e940da6d-ee8d-42f1-bdf2-e580a3705fab","Type":"ContainerStarted","Data":"2c07f2eca3e0db491dfdbdf06775cb2f208fdca8d61df01bcea23b4034362161"}
Nov 03 08:27:27.086030 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:27.085988    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5" podStartSLOduration=2.085975543 podStartE2EDuration="2.085975543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:27:27.085344435 +0000 UTC m=+2045.199808367" watchObservedRunningTime="2026-11-03 08:27:27.085975543 +0000 UTC m=+2045.200439471"
Nov 03 08:27:27.117442 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:27.117414    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-109070-ms2xd"]
Nov 03 08:27:27.117970 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:27:27.117950    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983d72df_19d6_4506_9794_ef75dbe012c3.slice/crio-b048ec7fa6427312ada6c420bbc23a6f3abc75cd3c701dd8e408b372ff6eae10 WatchSource:0}: Error finding container b048ec7fa6427312ada6c420bbc23a6f3abc75cd3c701dd8e408b372ff6eae10: Status 404 returned error can't find the container with id b048ec7fa6427312ada6c420bbc23a6f3abc75cd3c701dd8e408b372ff6eae10
Nov 03 08:27:28.079732 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:28.079652    5070 generic.go:356] "Generic (PLEG): container finished" podID="983d72df-19d6-4506-9794-ef75dbe012c3" containerID="6a513ac968a98bf31250dd1c015f173cec39a0989123bb402b89a08d7bdd1f48" exitCode=0
Nov 03 08:27:28.080130 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:28.079734    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-109070-ms2xd" event={"ID":"983d72df-19d6-4506-9794-ef75dbe012c3","Type":"ContainerDied","Data":"6a513ac968a98bf31250dd1c015f173cec39a0989123bb402b89a08d7bdd1f48"}
Nov 03 08:27:28.080130 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:28.079796    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-109070-ms2xd" event={"ID":"983d72df-19d6-4506-9794-ef75dbe012c3","Type":"ContainerStarted","Data":"b048ec7fa6427312ada6c420bbc23a6f3abc75cd3c701dd8e408b372ff6eae10"}
Nov 03 08:27:28.081324 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:28.081297    5070 generic.go:356] "Generic (PLEG): container finished" podID="e940da6d-ee8d-42f1-bdf2-e580a3705fab" containerID="d6f2c6e93cc28712da8e521fba2ef4cb373740f44f26a4cb9320f6cbf7556ce7" exitCode=0
Nov 03 08:27:28.081427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:28.081367    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5" event={"ID":"e940da6d-ee8d-42f1-bdf2-e580a3705fab","Type":"ContainerDied","Data":"d6f2c6e93cc28712da8e521fba2ef4cb373740f44f26a4cb9320f6cbf7556ce7"}
Nov 03 08:27:29.297941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.297918    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5"
Nov 03 08:27:29.301186 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.301164    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-109070-ms2xd"
Nov 03 08:27:29.370291 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.370260    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/983d72df-19d6-4506-9794-ef75dbe012c3-kube-api-access-6zhv8\" (UniqueName: \"kubernetes.io/projected/983d72df-19d6-4506-9794-ef75dbe012c3-kube-api-access-6zhv8\") pod \"983d72df-19d6-4506-9794-ef75dbe012c3\" (UID: \"983d72df-19d6-4506-9794-ef75dbe012c3\") "
Nov 03 08:27:29.370445 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.370404    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/983d72df-19d6-4506-9794-ef75dbe012c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983d72df-19d6-4506-9794-ef75dbe012c3-maas-api-openshift-ca\") pod \"983d72df-19d6-4506-9794-ef75dbe012c3\" (UID: \"983d72df-19d6-4506-9794-ef75dbe012c3\") "
Nov 03 08:27:29.370519 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.370452    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e940da6d-ee8d-42f1-bdf2-e580a3705fab-kube-api-access-t585r\" (UniqueName: \"kubernetes.io/projected/e940da6d-ee8d-42f1-bdf2-e580a3705fab-kube-api-access-t585r\") pod \"e940da6d-ee8d-42f1-bdf2-e580a3705fab\" (UID: \"e940da6d-ee8d-42f1-bdf2-e580a3705fab\") "
Nov 03 08:27:29.370584 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.370533    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e940da6d-ee8d-42f1-bdf2-e580a3705fab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e940da6d-ee8d-42f1-bdf2-e580a3705fab-maas-api-xks-ca\") pod \"e940da6d-ee8d-42f1-bdf2-e580a3705fab\" (UID: \"e940da6d-ee8d-42f1-bdf2-e580a3705fab\") "
Nov 03 08:27:29.370653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.370579    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e940da6d-ee8d-42f1-bdf2-e580a3705fab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e940da6d-ee8d-42f1-bdf2-e580a3705fab-maas-api-openshift-ca\") pod \"e940da6d-ee8d-42f1-bdf2-e580a3705fab\" (UID: \"e940da6d-ee8d-42f1-bdf2-e580a3705fab\") "
Nov 03 08:27:29.370653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.370612    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/983d72df-19d6-4506-9794-ef75dbe012c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983d72df-19d6-4506-9794-ef75dbe012c3-maas-api-xks-ca\") pod \"983d72df-19d6-4506-9794-ef75dbe012c3\" (UID: \"983d72df-19d6-4506-9794-ef75dbe012c3\") "
Nov 03 08:27:29.370851 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.370824    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/983d72df-19d6-4506-9794-ef75dbe012c3-maas-api-openshift-ca" pod "983d72df-19d6-4506-9794-ef75dbe012c3" (UID: "983d72df-19d6-4506-9794-ef75dbe012c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:27:29.371137 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.371109    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e940da6d-ee8d-42f1-bdf2-e580a3705fab-maas-api-openshift-ca" pod "e940da6d-ee8d-42f1-bdf2-e580a3705fab" (UID: "e940da6d-ee8d-42f1-bdf2-e580a3705fab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:27:29.371281 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.371261    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/983d72df-19d6-4506-9794-ef75dbe012c3-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:29.371281 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.371287    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e940da6d-ee8d-42f1-bdf2-e580a3705fab-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:29.372788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.372728    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/983d72df-19d6-4506-9794-ef75dbe012c3-kube-api-access-6zhv8" pod "983d72df-19d6-4506-9794-ef75dbe012c3" (UID: "983d72df-19d6-4506-9794-ef75dbe012c3"). InnerVolumeSpecName "kube-api-access-6zhv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:29.372901 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.372851    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/983d72df-19d6-4506-9794-ef75dbe012c3-maas-api-xks-ca" pod "983d72df-19d6-4506-9794-ef75dbe012c3" (UID: "983d72df-19d6-4506-9794-ef75dbe012c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:29.373007 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.372986    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e940da6d-ee8d-42f1-bdf2-e580a3705fab-kube-api-access-t585r" pod "e940da6d-ee8d-42f1-bdf2-e580a3705fab" (UID: "e940da6d-ee8d-42f1-bdf2-e580a3705fab"). InnerVolumeSpecName "kube-api-access-t585r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:29.373414 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.373394    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e940da6d-ee8d-42f1-bdf2-e580a3705fab-maas-api-xks-ca" pod "e940da6d-ee8d-42f1-bdf2-e580a3705fab" (UID: "e940da6d-ee8d-42f1-bdf2-e580a3705fab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:29.472468 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.472429    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t585r\" (UniqueName: \"kubernetes.io/projected/e940da6d-ee8d-42f1-bdf2-e580a3705fab-kube-api-access-t585r\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:29.472468 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.472461    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e940da6d-ee8d-42f1-bdf2-e580a3705fab-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:29.472468 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.472471    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/983d72df-19d6-4506-9794-ef75dbe012c3-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:29.472719 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.472480    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zhv8\" (UniqueName: \"kubernetes.io/projected/983d72df-19d6-4506-9794-ef75dbe012c3-kube-api-access-6zhv8\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:29.534949 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.534916    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s" podUID="84094812-6bd0-401a-a91c-d3d24e71501f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:27:29.535667 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:29.535638    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq" podUID="6b7840f4-acec-4021-9ab3-c15fc29f5f8b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:27:30.099063 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:30.099039    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-109070-ms2xd"
Nov 03 08:27:30.099207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:30.099082    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-109070-ms2xd" event={"ID":"983d72df-19d6-4506-9794-ef75dbe012c3","Type":"ContainerDied","Data":"b048ec7fa6427312ada6c420bbc23a6f3abc75cd3c701dd8e408b372ff6eae10"}
Nov 03 08:27:30.099207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:30.099128    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b048ec7fa6427312ada6c420bbc23a6f3abc75cd3c701dd8e408b372ff6eae10"
Nov 03 08:27:30.100633 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:30.100614    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5"
Nov 03 08:27:30.100776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:30.100649    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5" event={"ID":"e940da6d-ee8d-42f1-bdf2-e580a3705fab","Type":"ContainerDied","Data":"2c07f2eca3e0db491dfdbdf06775cb2f208fdca8d61df01bcea23b4034362161"}
Nov 03 08:27:30.100776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:30.100694    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c07f2eca3e0db491dfdbdf06775cb2f208fdca8d61df01bcea23b4034362161"
Nov 03 08:27:33.230102 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.230069    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp"]
Nov 03 08:27:33.230669 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.230656    5070 cpu_manager.go:414] "containerMap: removing container" podUID="e940da6d-ee8d-42f1-bdf2-e580a3705fab" containerName="revoke-keys"
Nov 03 08:27:33.230717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.230670    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e940da6d-ee8d-42f1-bdf2-e580a3705fab" containerName="revoke-keys"
Nov 03 08:27:33.230717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.230705    5070 cpu_manager.go:414] "containerMap: removing container" podUID="983d72df-19d6-4506-9794-ef75dbe012c3" containerName="revoke-keys"
Nov 03 08:27:33.230717 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.230711    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="983d72df-19d6-4506-9794-ef75dbe012c3" containerName="revoke-keys"
Nov 03 08:27:33.230838 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.230818    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="983d72df-19d6-4506-9794-ef75dbe012c3" containerName="revoke-keys"
Nov 03 08:27:33.230838 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.230833    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="e940da6d-ee8d-42f1-bdf2-e580a3705fab" containerName="revoke-keys"
Nov 03 08:27:33.235638 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.235618    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp"
Nov 03 08:27:33.240803 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.238911    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:27:33.240803 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.238982    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:27:33.244528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.244501    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp"]
Nov 03 08:27:33.276483 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.276447    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:27:33.276483 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.276468    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:27:33.335187 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.335156    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d646a8-437f-44a9-861a-a704958b3e12-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp\" (UID: \"c4d646a8-437f-44a9-861a-a704958b3e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp"
Nov 03 08:27:33.335353 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.335307    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrsml\" (UniqueName: \"kubernetes.io/projected/c4d646a8-437f-44a9-861a-a704958b3e12-kube-api-access-xrsml\") pod \"maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp\" (UID: \"c4d646a8-437f-44a9-861a-a704958b3e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp"
Nov 03 08:27:33.335410 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.335384    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d646a8-437f-44a9-861a-a704958b3e12-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp\" (UID: \"c4d646a8-437f-44a9-861a-a704958b3e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp"
Nov 03 08:27:33.436154 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.436124    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrsml\" (UniqueName: \"kubernetes.io/projected/c4d646a8-437f-44a9-861a-a704958b3e12-kube-api-access-xrsml\") pod \"maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp\" (UID: \"c4d646a8-437f-44a9-861a-a704958b3e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp"
Nov 03 08:27:33.436300 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.436159    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d646a8-437f-44a9-861a-a704958b3e12-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp\" (UID: \"c4d646a8-437f-44a9-861a-a704958b3e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp"
Nov 03 08:27:33.436300 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.436252    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d646a8-437f-44a9-861a-a704958b3e12-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp\" (UID: \"c4d646a8-437f-44a9-861a-a704958b3e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp"
Nov 03 08:27:33.436847 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.436817    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d646a8-437f-44a9-861a-a704958b3e12-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp\" (UID: \"c4d646a8-437f-44a9-861a-a704958b3e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp"
Nov 03 08:27:33.438725 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.438702    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d646a8-437f-44a9-861a-a704958b3e12-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp\" (UID: \"c4d646a8-437f-44a9-861a-a704958b3e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp"
Nov 03 08:27:33.443103 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.443084    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrsml\" (UniqueName: \"kubernetes.io/projected/c4d646a8-437f-44a9-861a-a704958b3e12-kube-api-access-xrsml\") pod \"maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp\" (UID: \"c4d646a8-437f-44a9-861a-a704958b3e12\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp"
Nov 03 08:27:33.554797 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.554713    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp"
Nov 03 08:27:33.838782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:33.834594    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp"]
Nov 03 08:27:34.133410 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:34.133375    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp" event={"ID":"c4d646a8-437f-44a9-861a-a704958b3e12","Type":"ContainerStarted","Data":"1402d9213a5467a0d29cc2641d10befed230a15ad683044430ed53866aa8495f"}
Nov 03 08:27:35.144856 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.144828    5070 generic.go:356] "Generic (PLEG): container finished" podID="c4d646a8-437f-44a9-861a-a704958b3e12" containerID="12590141bcff365cdfdf6ed9de32d4570b73fe7b8c8c086750413a608a1d482d" exitCode=0
Nov 03 08:27:35.145290 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.145242    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp" event={"ID":"c4d646a8-437f-44a9-861a-a704958b3e12","Type":"ContainerDied","Data":"12590141bcff365cdfdf6ed9de32d4570b73fe7b8c8c086750413a608a1d482d"}
Nov 03 08:27:35.146936 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.146915    5070 generic.go:356] "Generic (PLEG): container finished" podID="6b7840f4-acec-4021-9ab3-c15fc29f5f8b" containerID="79aca1a1021c1bb1cf4fc500127cb274a1d12f6a454e58edf89f65f133fbe158" exitCode=0
Nov 03 08:27:35.147069 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.146995    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq" event={"ID":"6b7840f4-acec-4021-9ab3-c15fc29f5f8b","Type":"ContainerDied","Data":"79aca1a1021c1bb1cf4fc500127cb274a1d12f6a454e58edf89f65f133fbe158"}
Nov 03 08:27:35.147069 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.147043    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq" event={"ID":"6b7840f4-acec-4021-9ab3-c15fc29f5f8b","Type":"ContainerDied","Data":"8c327a769b79d4d6f54895468952bdb53623289b57ff92a20e6d2e16ccdc2b4d"}
Nov 03 08:27:35.147069 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.147067    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c327a769b79d4d6f54895468952bdb53623289b57ff92a20e6d2e16ccdc2b4d"
Nov 03 08:27:35.148904 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.148881    5070 generic.go:356] "Generic (PLEG): container finished" podID="84094812-6bd0-401a-a91c-d3d24e71501f" containerID="4df1c0060b45187901e2d6d0a4ef244479c0b5de76163af428e0c148684ba023" exitCode=0
Nov 03 08:27:35.148980 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.148943    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s" event={"ID":"84094812-6bd0-401a-a91c-d3d24e71501f","Type":"ContainerDied","Data":"4df1c0060b45187901e2d6d0a4ef244479c0b5de76163af428e0c148684ba023"}
Nov 03 08:27:35.154519 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.154500    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq"
Nov 03 08:27:35.194848 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.194827    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s"
Nov 03 08:27:35.259245 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.259184    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b7840f4-acec-4021-9ab3-c15fc29f5f8b-kube-api-access-mq7qf\" (UniqueName: \"kubernetes.io/projected/6b7840f4-acec-4021-9ab3-c15fc29f5f8b-kube-api-access-mq7qf\") pod \"6b7840f4-acec-4021-9ab3-c15fc29f5f8b\" (UID: \"6b7840f4-acec-4021-9ab3-c15fc29f5f8b\") "
Nov 03 08:27:35.259245 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.259240    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84094812-6bd0-401a-a91c-d3d24e71501f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84094812-6bd0-401a-a91c-d3d24e71501f-plugins-config-volume\") pod \"84094812-6bd0-401a-a91c-d3d24e71501f\" (UID: \"84094812-6bd0-401a-a91c-d3d24e71501f\") "
Nov 03 08:27:35.259403 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.259267    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84094812-6bd0-401a-a91c-d3d24e71501f-kube-api-access-kpp2p\" (UniqueName: \"kubernetes.io/projected/84094812-6bd0-401a-a91c-d3d24e71501f-kube-api-access-kpp2p\") pod \"84094812-6bd0-401a-a91c-d3d24e71501f\" (UID: \"84094812-6bd0-401a-a91c-d3d24e71501f\") "
Nov 03 08:27:35.259403 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.259310    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7840f4-acec-4021-9ab3-c15fc29f5f8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7840f4-acec-4021-9ab3-c15fc29f5f8b-plugins-config-volume\") pod \"6b7840f4-acec-4021-9ab3-c15fc29f5f8b\" (UID: \"6b7840f4-acec-4021-9ab3-c15fc29f5f8b\") "
Nov 03 08:27:35.259713 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.259690    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84094812-6bd0-401a-a91c-d3d24e71501f-plugins-config-volume" pod "84094812-6bd0-401a-a91c-d3d24e71501f" (UID: "84094812-6bd0-401a-a91c-d3d24e71501f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:27:35.259835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.259727    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7840f4-acec-4021-9ab3-c15fc29f5f8b-plugins-config-volume" pod "6b7840f4-acec-4021-9ab3-c15fc29f5f8b" (UID: "6b7840f4-acec-4021-9ab3-c15fc29f5f8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:27:35.260057 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.260036    5070 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84094812-6bd0-401a-a91c-d3d24e71501f-plugins-config-volume\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:35.260112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.260063    5070 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b7840f4-acec-4021-9ab3-c15fc29f5f8b-plugins-config-volume\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:35.261339 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.261315    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7840f4-acec-4021-9ab3-c15fc29f5f8b-kube-api-access-mq7qf" pod "6b7840f4-acec-4021-9ab3-c15fc29f5f8b" (UID: "6b7840f4-acec-4021-9ab3-c15fc29f5f8b"). InnerVolumeSpecName "kube-api-access-mq7qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:35.261443 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.261348    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84094812-6bd0-401a-a91c-d3d24e71501f-kube-api-access-kpp2p" pod "84094812-6bd0-401a-a91c-d3d24e71501f" (UID: "84094812-6bd0-401a-a91c-d3d24e71501f"). InnerVolumeSpecName "kube-api-access-kpp2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:35.360976 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.360947    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq7qf\" (UniqueName: \"kubernetes.io/projected/6b7840f4-acec-4021-9ab3-c15fc29f5f8b-kube-api-access-mq7qf\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:35.360976 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:35.360968    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpp2p\" (UniqueName: \"kubernetes.io/projected/84094812-6bd0-401a-a91c-d3d24e71501f-kube-api-access-kpp2p\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:36.162561 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.162525    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s" event={"ID":"84094812-6bd0-401a-a91c-d3d24e71501f","Type":"ContainerDied","Data":"d4bc1da003ceb62e3d0ba571c06fdf384cf6b1c3b93e89e9ea38f110c63bed84"}
Nov 03 08:27:36.163109 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.162580    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s"
Nov 03 08:27:36.163109 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.162622    5070 scope.go:122] "RemoveContainer" containerID="4df1c0060b45187901e2d6d0a4ef244479c0b5de76163af428e0c148684ba023"
Nov 03 08:27:36.163109 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.162926    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq"
Nov 03 08:27:36.188324 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.188290    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s"]
Nov 03 08:27:36.194211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.194141    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a66d5c-74c768ddf9-9755s"]
Nov 03 08:27:36.213683 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.213657    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq"]
Nov 03 08:27:36.218636 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.218614    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a66d5c-565878d947-k7lhq"]
Nov 03 08:27:36.342134 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.340687    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp"
Nov 03 08:27:36.486906 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.485880    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4d646a8-437f-44a9-861a-a704958b3e12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d646a8-437f-44a9-861a-a704958b3e12-maas-api-xks-ca\") pod \"c4d646a8-437f-44a9-861a-a704958b3e12\" (UID: \"c4d646a8-437f-44a9-861a-a704958b3e12\") "
Nov 03 08:27:36.486906 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.486057    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4d646a8-437f-44a9-861a-a704958b3e12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d646a8-437f-44a9-861a-a704958b3e12-maas-api-openshift-ca\") pod \"c4d646a8-437f-44a9-861a-a704958b3e12\" (UID: \"c4d646a8-437f-44a9-861a-a704958b3e12\") "
Nov 03 08:27:36.486906 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.486099    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4d646a8-437f-44a9-861a-a704958b3e12-kube-api-access-xrsml\" (UniqueName: \"kubernetes.io/projected/c4d646a8-437f-44a9-861a-a704958b3e12-kube-api-access-xrsml\") pod \"c4d646a8-437f-44a9-861a-a704958b3e12\" (UID: \"c4d646a8-437f-44a9-861a-a704958b3e12\") "
Nov 03 08:27:36.487661 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.487630    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d646a8-437f-44a9-861a-a704958b3e12-maas-api-openshift-ca" pod "c4d646a8-437f-44a9-861a-a704958b3e12" (UID: "c4d646a8-437f-44a9-861a-a704958b3e12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:27:36.491300 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.491274    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4d646a8-437f-44a9-861a-a704958b3e12-maas-api-xks-ca" pod "c4d646a8-437f-44a9-861a-a704958b3e12" (UID: "c4d646a8-437f-44a9-861a-a704958b3e12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:36.491300 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.491279    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d646a8-437f-44a9-861a-a704958b3e12-kube-api-access-xrsml" pod "c4d646a8-437f-44a9-861a-a704958b3e12" (UID: "c4d646a8-437f-44a9-861a-a704958b3e12"). InnerVolumeSpecName "kube-api-access-xrsml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:36.587371 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.587303    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d646a8-437f-44a9-861a-a704958b3e12-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:36.587371 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.587330    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d646a8-437f-44a9-861a-a704958b3e12-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:36.587371 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.587348    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrsml\" (UniqueName: \"kubernetes.io/projected/c4d646a8-437f-44a9-861a-a704958b3e12-kube-api-access-xrsml\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:36.596516 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.596484    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ccb783-pg7wz"]
Nov 03 08:27:36.597052 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.597037    5070 cpu_manager.go:414] "containerMap: removing container" podUID="84094812-6bd0-401a-a91c-d3d24e71501f" containerName="payload-pre-processing"
Nov 03 08:27:36.597100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.597053    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84094812-6bd0-401a-a91c-d3d24e71501f" containerName="payload-pre-processing"
Nov 03 08:27:36.597100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.597088    5070 cpu_manager.go:414] "containerMap: removing container" podUID="6b7840f4-acec-4021-9ab3-c15fc29f5f8b" containerName="payload-processing"
Nov 03 08:27:36.597100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.597094    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7840f4-acec-4021-9ab3-c15fc29f5f8b" containerName="payload-processing"
Nov 03 08:27:36.597196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.597130    5070 cpu_manager.go:414] "containerMap: removing container" podUID="c4d646a8-437f-44a9-861a-a704958b3e12" containerName="revoke-keys"
Nov 03 08:27:36.597196 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.597135    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4d646a8-437f-44a9-861a-a704958b3e12" containerName="revoke-keys"
Nov 03 08:27:36.597265 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.597216    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7840f4-acec-4021-9ab3-c15fc29f5f8b" containerName="payload-processing"
Nov 03 08:27:36.597265 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.597231    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4d646a8-437f-44a9-861a-a704958b3e12" containerName="revoke-keys"
Nov 03 08:27:36.597265 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.597240    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="84094812-6bd0-401a-a91c-d3d24e71501f" containerName="payload-pre-processing"
Nov 03 08:27:36.600843 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.600827    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ccb783-pg7wz"
Nov 03 08:27:36.623695 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.623668    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ccb783-pg7wz"]
Nov 03 08:27:36.688773 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.688737    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1fa3109-88e6-427c-8239-f1f1604231f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1ccb783-pg7wz\" (UID: \"c1fa3109-88e6-427c-8239-f1f1604231f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ccb783-pg7wz"
Nov 03 08:27:36.688894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.688848    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1fa3109-88e6-427c-8239-f1f1604231f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1ccb783-pg7wz\" (UID: \"c1fa3109-88e6-427c-8239-f1f1604231f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ccb783-pg7wz"
Nov 03 08:27:36.688894 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.688876    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l44zp\" (UniqueName: \"kubernetes.io/projected/c1fa3109-88e6-427c-8239-f1f1604231f9-kube-api-access-l44zp\") pod \"maas-api-revoke-sub-e1ccb783-pg7wz\" (UID: \"c1fa3109-88e6-427c-8239-f1f1604231f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ccb783-pg7wz"
Nov 03 08:27:36.790386 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.790353    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1fa3109-88e6-427c-8239-f1f1604231f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1ccb783-pg7wz\" (UID: \"c1fa3109-88e6-427c-8239-f1f1604231f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ccb783-pg7wz"
Nov 03 08:27:36.790589 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.790452    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1fa3109-88e6-427c-8239-f1f1604231f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1ccb783-pg7wz\" (UID: \"c1fa3109-88e6-427c-8239-f1f1604231f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ccb783-pg7wz"
Nov 03 08:27:36.790589 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.790485    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l44zp\" (UniqueName: \"kubernetes.io/projected/c1fa3109-88e6-427c-8239-f1f1604231f9-kube-api-access-l44zp\") pod \"maas-api-revoke-sub-e1ccb783-pg7wz\" (UID: \"c1fa3109-88e6-427c-8239-f1f1604231f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ccb783-pg7wz"
Nov 03 08:27:36.791209 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.791185    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1fa3109-88e6-427c-8239-f1f1604231f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1ccb783-pg7wz\" (UID: \"c1fa3109-88e6-427c-8239-f1f1604231f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ccb783-pg7wz"
Nov 03 08:27:36.793498 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.793474    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1fa3109-88e6-427c-8239-f1f1604231f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1ccb783-pg7wz\" (UID: \"c1fa3109-88e6-427c-8239-f1f1604231f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ccb783-pg7wz"
Nov 03 08:27:36.800653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.800631    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l44zp\" (UniqueName: \"kubernetes.io/projected/c1fa3109-88e6-427c-8239-f1f1604231f9-kube-api-access-l44zp\") pod \"maas-api-revoke-sub-e1ccb783-pg7wz\" (UID: \"c1fa3109-88e6-427c-8239-f1f1604231f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ccb783-pg7wz"
Nov 03 08:27:36.915527 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:36.915497    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ccb783-pg7wz"
Nov 03 08:27:37.172687 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:37.172607    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp"
Nov 03 08:27:37.172687 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:37.172644    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp" event={"ID":"c4d646a8-437f-44a9-861a-a704958b3e12","Type":"ContainerDied","Data":"1402d9213a5467a0d29cc2641d10befed230a15ad683044430ed53866aa8495f"}
Nov 03 08:27:37.173226 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:37.172693    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1402d9213a5467a0d29cc2641d10befed230a15ad683044430ed53866aa8495f"
Nov 03 08:27:37.200660 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:37.200631    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ccb783-pg7wz"]
Nov 03 08:27:37.200872 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:27:37.200850    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1fa3109_88e6_427c_8239_f1f1604231f9.slice/crio-db8d5d0796b00683fe8c0736314a737cb56aaba13730cefbe0767f6999dd40a4 WatchSource:0}: Error finding container db8d5d0796b00683fe8c0736314a737cb56aaba13730cefbe0767f6999dd40a4: Status 404 returned error can't find the container with id db8d5d0796b00683fe8c0736314a737cb56aaba13730cefbe0767f6999dd40a4
Nov 03 08:27:38.025743 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:38.025710    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b7840f4-acec-4021-9ab3-c15fc29f5f8b" path="/var/lib/kubelet/pods/6b7840f4-acec-4021-9ab3-c15fc29f5f8b/volumes"
Nov 03 08:27:38.026167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:38.026152    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84094812-6bd0-401a-a91c-d3d24e71501f" path="/var/lib/kubelet/pods/84094812-6bd0-401a-a91c-d3d24e71501f/volumes"
Nov 03 08:27:38.048138 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:38.048112    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"]
Nov 03 08:27:38.070421 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:38.070395    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"]
Nov 03 08:27:38.184174 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:38.184143    5070 generic.go:356] "Generic (PLEG): container finished" podID="c1fa3109-88e6-427c-8239-f1f1604231f9" containerID="3b0319880bd4d244a1bf16da1c2705ea21cc4bd41538bc149bc90aa456a68a5a" exitCode=0
Nov 03 08:27:38.184510 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:38.184227    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ccb783-pg7wz" event={"ID":"c1fa3109-88e6-427c-8239-f1f1604231f9","Type":"ContainerDied","Data":"3b0319880bd4d244a1bf16da1c2705ea21cc4bd41538bc149bc90aa456a68a5a"}
Nov 03 08:27:38.184510 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:38.184275    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ccb783-pg7wz" event={"ID":"c1fa3109-88e6-427c-8239-f1f1604231f9","Type":"ContainerStarted","Data":"db8d5d0796b00683fe8c0736314a737cb56aaba13730cefbe0767f6999dd40a4"}
Nov 03 08:27:39.356123 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:39.356099    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ccb783-pg7wz"
Nov 03 08:27:39.423262 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:39.423227    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1fa3109-88e6-427c-8239-f1f1604231f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1fa3109-88e6-427c-8239-f1f1604231f9-maas-api-openshift-ca\") pod \"c1fa3109-88e6-427c-8239-f1f1604231f9\" (UID: \"c1fa3109-88e6-427c-8239-f1f1604231f9\") "
Nov 03 08:27:39.423438 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:39.423367    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1fa3109-88e6-427c-8239-f1f1604231f9-kube-api-access-l44zp\" (UniqueName: \"kubernetes.io/projected/c1fa3109-88e6-427c-8239-f1f1604231f9-kube-api-access-l44zp\") pod \"c1fa3109-88e6-427c-8239-f1f1604231f9\" (UID: \"c1fa3109-88e6-427c-8239-f1f1604231f9\") "
Nov 03 08:27:39.423518 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:39.423445    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1fa3109-88e6-427c-8239-f1f1604231f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1fa3109-88e6-427c-8239-f1f1604231f9-maas-api-xks-ca\") pod \"c1fa3109-88e6-427c-8239-f1f1604231f9\" (UID: \"c1fa3109-88e6-427c-8239-f1f1604231f9\") "
Nov 03 08:27:39.423649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:39.423625    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1fa3109-88e6-427c-8239-f1f1604231f9-maas-api-openshift-ca" pod "c1fa3109-88e6-427c-8239-f1f1604231f9" (UID: "c1fa3109-88e6-427c-8239-f1f1604231f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:27:39.423914 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:39.423895    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1fa3109-88e6-427c-8239-f1f1604231f9-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:39.425588 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:39.425563    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1fa3109-88e6-427c-8239-f1f1604231f9-kube-api-access-l44zp" pod "c1fa3109-88e6-427c-8239-f1f1604231f9" (UID: "c1fa3109-88e6-427c-8239-f1f1604231f9"). InnerVolumeSpecName "kube-api-access-l44zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:39.426111 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:39.426091    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1fa3109-88e6-427c-8239-f1f1604231f9-maas-api-xks-ca" pod "c1fa3109-88e6-427c-8239-f1f1604231f9" (UID: "c1fa3109-88e6-427c-8239-f1f1604231f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:39.524874 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:39.524821    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l44zp\" (UniqueName: \"kubernetes.io/projected/c1fa3109-88e6-427c-8239-f1f1604231f9-kube-api-access-l44zp\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:39.524874 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:39.524846    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1fa3109-88e6-427c-8239-f1f1604231f9-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:40.202393 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:40.202354    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ccb783-pg7wz" event={"ID":"c1fa3109-88e6-427c-8239-f1f1604231f9","Type":"ContainerDied","Data":"db8d5d0796b00683fe8c0736314a737cb56aaba13730cefbe0767f6999dd40a4"}
Nov 03 08:27:40.202393 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:40.202374    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ccb783-pg7wz"
Nov 03 08:27:40.202605 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:40.202410    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db8d5d0796b00683fe8c0736314a737cb56aaba13730cefbe0767f6999dd40a4"
Nov 03 08:27:43.365000 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:43.364968    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg"]
Nov 03 08:27:43.926013 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:43.925973    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-109070-ms2xd"]
Nov 03 08:27:43.929510 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:43.929482    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-109070-ms2xd"]
Nov 03 08:27:44.025547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:44.025511    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="983d72df-19d6-4506-9794-ef75dbe012c3" path="/var/lib/kubelet/pods/983d72df-19d6-4506-9794-ef75dbe012c3/volumes"
Nov 03 08:27:45.016564 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.016531    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:27:45.613161 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.613127    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5"]
Nov 03 08:27:45.613686 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.613672    5070 cpu_manager.go:414] "containerMap: removing container" podUID="c1fa3109-88e6-427c-8239-f1f1604231f9" containerName="revoke-keys"
Nov 03 08:27:45.613734 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.613687    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1fa3109-88e6-427c-8239-f1f1604231f9" containerName="revoke-keys"
Nov 03 08:27:45.613843 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.613832    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1fa3109-88e6-427c-8239-f1f1604231f9" containerName="revoke-keys"
Nov 03 08:27:45.617274 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.617250    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5"
Nov 03 08:27:45.622584 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.622560    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b929697f-6btsc"]
Nov 03 08:27:45.627295 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.627275    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b929697f-6btsc"
Nov 03 08:27:45.628612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.628590    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:27:45.648670 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.630850    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:27:45.648797 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.630898    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:27:45.648885 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.630951    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:27:45.697630 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.697602    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5"]
Nov 03 08:27:45.700208 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.700183    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7eb5a76-5765-468d-972b-08ce645c4fa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a1c6b34-ggnf5\" (UID: \"c7eb5a76-5765-468d-972b-08ce645c4fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5"
Nov 03 08:27:45.700350 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.700224    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jbsx\" (UniqueName: \"kubernetes.io/projected/c7eb5a76-5765-468d-972b-08ce645c4fa8-kube-api-access-8jbsx\") pod \"maas-api-revoke-sub-4a1c6b34-ggnf5\" (UID: \"c7eb5a76-5765-468d-972b-08ce645c4fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5"
Nov 03 08:27:45.700404 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.700383    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7eb5a76-5765-468d-972b-08ce645c4fa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a1c6b34-ggnf5\" (UID: \"c7eb5a76-5765-468d-972b-08ce645c4fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5"
Nov 03 08:27:45.704042 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.704022    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b929697f-6btsc"]
Nov 03 08:27:45.802647 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.802610    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7eb5a76-5765-468d-972b-08ce645c4fa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a1c6b34-ggnf5\" (UID: \"c7eb5a76-5765-468d-972b-08ce645c4fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5"
Nov 03 08:27:45.805811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.805778    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55nnw\" (UniqueName: \"kubernetes.io/projected/e569b99d-12a2-455b-9f87-45e229768318-kube-api-access-55nnw\") pod \"maas-api-revoke-sub-b929697f-6btsc\" (UID: \"e569b99d-12a2-455b-9f87-45e229768318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b929697f-6btsc"
Nov 03 08:27:45.813749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.811011    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e569b99d-12a2-455b-9f87-45e229768318-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b929697f-6btsc\" (UID: \"e569b99d-12a2-455b-9f87-45e229768318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b929697f-6btsc"
Nov 03 08:27:45.813749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.811165    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7eb5a76-5765-468d-972b-08ce645c4fa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a1c6b34-ggnf5\" (UID: \"c7eb5a76-5765-468d-972b-08ce645c4fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5"
Nov 03 08:27:45.813749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.811198    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jbsx\" (UniqueName: \"kubernetes.io/projected/c7eb5a76-5765-468d-972b-08ce645c4fa8-kube-api-access-8jbsx\") pod \"maas-api-revoke-sub-4a1c6b34-ggnf5\" (UID: \"c7eb5a76-5765-468d-972b-08ce645c4fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5"
Nov 03 08:27:45.813749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.811292    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e569b99d-12a2-455b-9f87-45e229768318-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b929697f-6btsc\" (UID: \"e569b99d-12a2-455b-9f87-45e229768318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b929697f-6btsc"
Nov 03 08:27:45.813749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.804660    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7eb5a76-5765-468d-972b-08ce645c4fa8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a1c6b34-ggnf5\" (UID: \"c7eb5a76-5765-468d-972b-08ce645c4fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5"
Nov 03 08:27:45.820317 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.820294    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jbsx\" (UniqueName: \"kubernetes.io/projected/c7eb5a76-5765-468d-972b-08ce645c4fa8-kube-api-access-8jbsx\") pod \"maas-api-revoke-sub-4a1c6b34-ggnf5\" (UID: \"c7eb5a76-5765-468d-972b-08ce645c4fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5"
Nov 03 08:27:45.820498 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.820452    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7eb5a76-5765-468d-972b-08ce645c4fa8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a1c6b34-ggnf5\" (UID: \"c7eb5a76-5765-468d-972b-08ce645c4fa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5"
Nov 03 08:27:45.912912 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.912799    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e569b99d-12a2-455b-9f87-45e229768318-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b929697f-6btsc\" (UID: \"e569b99d-12a2-455b-9f87-45e229768318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b929697f-6btsc"
Nov 03 08:27:45.913094 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.912967    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55nnw\" (UniqueName: \"kubernetes.io/projected/e569b99d-12a2-455b-9f87-45e229768318-kube-api-access-55nnw\") pod \"maas-api-revoke-sub-b929697f-6btsc\" (UID: \"e569b99d-12a2-455b-9f87-45e229768318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b929697f-6btsc"
Nov 03 08:27:45.913094 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.913005    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e569b99d-12a2-455b-9f87-45e229768318-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b929697f-6btsc\" (UID: \"e569b99d-12a2-455b-9f87-45e229768318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b929697f-6btsc"
Nov 03 08:27:45.913573 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.913545    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e569b99d-12a2-455b-9f87-45e229768318-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b929697f-6btsc\" (UID: \"e569b99d-12a2-455b-9f87-45e229768318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b929697f-6btsc"
Nov 03 08:27:45.915634 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.915599    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e569b99d-12a2-455b-9f87-45e229768318-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b929697f-6btsc\" (UID: \"e569b99d-12a2-455b-9f87-45e229768318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b929697f-6btsc"
Nov 03 08:27:45.924671 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.924650    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55nnw\" (UniqueName: \"kubernetes.io/projected/e569b99d-12a2-455b-9f87-45e229768318-kube-api-access-55nnw\") pod \"maas-api-revoke-sub-b929697f-6btsc\" (UID: \"e569b99d-12a2-455b-9f87-45e229768318\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b929697f-6btsc"
Nov 03 08:27:45.953678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.953649    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5"
Nov 03 08:27:45.980172 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:45.979977    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b929697f-6btsc"
Nov 03 08:27:46.262711 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:46.262682    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5"]
Nov 03 08:27:46.263187 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:27:46.263021    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7eb5a76_5765_468d_972b_08ce645c4fa8.slice/crio-33ca663a428d9c534f8455eb6307a920cca1cf1626df4d7e5d6d6b7b8c9484f8 WatchSource:0}: Error finding container 33ca663a428d9c534f8455eb6307a920cca1cf1626df4d7e5d6d6b7b8c9484f8: Status 404 returned error can't find the container with id 33ca663a428d9c534f8455eb6307a920cca1cf1626df4d7e5d6d6b7b8c9484f8
Nov 03 08:27:46.278983 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:46.278956    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b929697f-6btsc"]
Nov 03 08:27:46.280422 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:27:46.280399    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode569b99d_12a2_455b_9f87_45e229768318.slice/crio-8f7243f9ba3d2c98bb50736b0c073d527d04cc5b3eaa4c2c3c227d33262892a5 WatchSource:0}: Error finding container 8f7243f9ba3d2c98bb50736b0c073d527d04cc5b3eaa4c2c3c227d33262892a5: Status 404 returned error can't find the container with id 8f7243f9ba3d2c98bb50736b0c073d527d04cc5b3eaa4c2c3c227d33262892a5
Nov 03 08:27:47.263075 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:47.263041    5070 generic.go:356] "Generic (PLEG): container finished" podID="e569b99d-12a2-455b-9f87-45e229768318" containerID="9eac18a25fa32a547926a446e24a0c0abd72a19e09b37490d8576f60dad28b21" exitCode=0
Nov 03 08:27:47.263448 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:47.263119    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b929697f-6btsc" event={"ID":"e569b99d-12a2-455b-9f87-45e229768318","Type":"ContainerDied","Data":"9eac18a25fa32a547926a446e24a0c0abd72a19e09b37490d8576f60dad28b21"}
Nov 03 08:27:47.263448 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:47.263169    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b929697f-6btsc" event={"ID":"e569b99d-12a2-455b-9f87-45e229768318","Type":"ContainerStarted","Data":"8f7243f9ba3d2c98bb50736b0c073d527d04cc5b3eaa4c2c3c227d33262892a5"}
Nov 03 08:27:47.265188 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:47.265163    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5" event={"ID":"c7eb5a76-5765-468d-972b-08ce645c4fa8","Type":"ContainerStarted","Data":"418c79eb23f708b0c8c3a86433b336a2384f4059602fd85d31d0e596009ae455"}
Nov 03 08:27:47.265304 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:47.265217    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5" event={"ID":"c7eb5a76-5765-468d-972b-08ce645c4fa8","Type":"ContainerStarted","Data":"33ca663a428d9c534f8455eb6307a920cca1cf1626df4d7e5d6d6b7b8c9484f8"}
Nov 03 08:27:47.317189 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:47.317138    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5" podStartSLOduration=2.317124113 podStartE2EDuration="2.317124113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:27:47.314644888 +0000 UTC m=+2065.429108820" watchObservedRunningTime="2026-11-03 08:27:47.317124113 +0000 UTC m=+2065.431588044"
Nov 03 08:27:47.558639 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:47.558563    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq"]
Nov 03 08:27:47.562719 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:47.562699    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq"
Nov 03 08:27:47.583733 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:47.583699    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq"]
Nov 03 08:27:47.635450 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:47.635417    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98x27\" (UniqueName: \"kubernetes.io/projected/63e7b26a-4296-4429-9112-2e47f81773b6-kube-api-access-98x27\") pod \"maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq\" (UID: \"63e7b26a-4296-4429-9112-2e47f81773b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq"
Nov 03 08:27:47.635661 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:47.635496    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e7b26a-4296-4429-9112-2e47f81773b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq\" (UID: \"63e7b26a-4296-4429-9112-2e47f81773b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq"
Nov 03 08:27:47.635745 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:47.635671    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e7b26a-4296-4429-9112-2e47f81773b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq\" (UID: \"63e7b26a-4296-4429-9112-2e47f81773b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq"
Nov 03 08:27:47.736772 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:47.736726    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e7b26a-4296-4429-9112-2e47f81773b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq\" (UID: \"63e7b26a-4296-4429-9112-2e47f81773b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq"
Nov 03 08:27:47.736932 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:47.736888    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98x27\" (UniqueName: \"kubernetes.io/projected/63e7b26a-4296-4429-9112-2e47f81773b6-kube-api-access-98x27\") pod \"maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq\" (UID: \"63e7b26a-4296-4429-9112-2e47f81773b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq"
Nov 03 08:27:47.737000 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:47.736947    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e7b26a-4296-4429-9112-2e47f81773b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq\" (UID: \"63e7b26a-4296-4429-9112-2e47f81773b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq"
Nov 03 08:27:47.737575 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:47.737553    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e7b26a-4296-4429-9112-2e47f81773b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq\" (UID: \"63e7b26a-4296-4429-9112-2e47f81773b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq"
Nov 03 08:27:47.741155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:47.741123    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e7b26a-4296-4429-9112-2e47f81773b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq\" (UID: \"63e7b26a-4296-4429-9112-2e47f81773b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq"
Nov 03 08:27:47.750258 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:47.750233    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98x27\" (UniqueName: \"kubernetes.io/projected/63e7b26a-4296-4429-9112-2e47f81773b6-kube-api-access-98x27\") pod \"maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq\" (UID: \"63e7b26a-4296-4429-9112-2e47f81773b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq"
Nov 03 08:27:47.876684 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:47.876658    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq"
Nov 03 08:27:48.210838 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:48.182559    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq"]
Nov 03 08:27:48.210838 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:27:48.182570    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e7b26a_4296_4429_9112_2e47f81773b6.slice/crio-1c4d0c562e3ac9bfe871953e9453cb58afa0711dcc1e29108ac80a6fd48ee4e5 WatchSource:0}: Error finding container 1c4d0c562e3ac9bfe871953e9453cb58afa0711dcc1e29108ac80a6fd48ee4e5: Status 404 returned error can't find the container with id 1c4d0c562e3ac9bfe871953e9453cb58afa0711dcc1e29108ac80a6fd48ee4e5
Nov 03 08:27:48.273994 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:48.273960    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq" event={"ID":"63e7b26a-4296-4429-9112-2e47f81773b6","Type":"ContainerStarted","Data":"1c4d0c562e3ac9bfe871953e9453cb58afa0711dcc1e29108ac80a6fd48ee4e5"}
Nov 03 08:27:48.275994 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:48.275969    5070 generic.go:356] "Generic (PLEG): container finished" podID="c7eb5a76-5765-468d-972b-08ce645c4fa8" containerID="418c79eb23f708b0c8c3a86433b336a2384f4059602fd85d31d0e596009ae455" exitCode=0
Nov 03 08:27:48.276133 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:48.276064    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5" event={"ID":"c7eb5a76-5765-468d-972b-08ce645c4fa8","Type":"ContainerDied","Data":"418c79eb23f708b0c8c3a86433b336a2384f4059602fd85d31d0e596009ae455"}
Nov 03 08:27:48.369502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:48.369440    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg" podUID="a2bafc12-e2f3-46ee-8f98-01e46c2c2787" containerName="payload-processing" containerID="cri-o://9a349cac83afa17a6fd7dcd2183a4c8615aecc6cdb9c5ff2a4897155383bcb1f" gracePeriod=25
Nov 03 08:27:48.437700 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:48.437646    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b929697f-6btsc"
Nov 03 08:27:48.546317 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:48.546286    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e569b99d-12a2-455b-9f87-45e229768318-kube-api-access-55nnw\" (UniqueName: \"kubernetes.io/projected/e569b99d-12a2-455b-9f87-45e229768318-kube-api-access-55nnw\") pod \"e569b99d-12a2-455b-9f87-45e229768318\" (UID: \"e569b99d-12a2-455b-9f87-45e229768318\") "
Nov 03 08:27:48.546489 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:48.546458    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e569b99d-12a2-455b-9f87-45e229768318-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e569b99d-12a2-455b-9f87-45e229768318-maas-api-openshift-ca\") pod \"e569b99d-12a2-455b-9f87-45e229768318\" (UID: \"e569b99d-12a2-455b-9f87-45e229768318\") "
Nov 03 08:27:48.546554 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:48.546519    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e569b99d-12a2-455b-9f87-45e229768318-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e569b99d-12a2-455b-9f87-45e229768318-maas-api-xks-ca\") pod \"e569b99d-12a2-455b-9f87-45e229768318\" (UID: \"e569b99d-12a2-455b-9f87-45e229768318\") "
Nov 03 08:27:48.546897 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:48.546856    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e569b99d-12a2-455b-9f87-45e229768318-maas-api-openshift-ca" pod "e569b99d-12a2-455b-9f87-45e229768318" (UID: "e569b99d-12a2-455b-9f87-45e229768318"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:27:48.547027 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:48.546972    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e569b99d-12a2-455b-9f87-45e229768318-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:48.556336 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:48.556312    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e569b99d-12a2-455b-9f87-45e229768318-maas-api-xks-ca" pod "e569b99d-12a2-455b-9f87-45e229768318" (UID: "e569b99d-12a2-455b-9f87-45e229768318"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:48.556462 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:48.556370    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e569b99d-12a2-455b-9f87-45e229768318-kube-api-access-55nnw" pod "e569b99d-12a2-455b-9f87-45e229768318" (UID: "e569b99d-12a2-455b-9f87-45e229768318"). InnerVolumeSpecName "kube-api-access-55nnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:48.648385 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:48.648361    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e569b99d-12a2-455b-9f87-45e229768318-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:48.648477 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:48.648387    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55nnw\" (UniqueName: \"kubernetes.io/projected/e569b99d-12a2-455b-9f87-45e229768318-kube-api-access-55nnw\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:49.285406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:49.285346    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b929697f-6btsc"
Nov 03 08:27:49.285777 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:49.285346    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b929697f-6btsc" event={"ID":"e569b99d-12a2-455b-9f87-45e229768318","Type":"ContainerDied","Data":"8f7243f9ba3d2c98bb50736b0c073d527d04cc5b3eaa4c2c3c227d33262892a5"}
Nov 03 08:27:49.285777 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:49.285473    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f7243f9ba3d2c98bb50736b0c073d527d04cc5b3eaa4c2c3c227d33262892a5"
Nov 03 08:27:49.287235 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:49.287208    5070 generic.go:356] "Generic (PLEG): container finished" podID="63e7b26a-4296-4429-9112-2e47f81773b6" containerID="f7234f2291356a29b53722e4fddf2edb419fb84c5344deb364587403a800ad31" exitCode=0
Nov 03 08:27:49.287361 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:49.287255    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq" event={"ID":"63e7b26a-4296-4429-9112-2e47f81773b6","Type":"ContainerDied","Data":"f7234f2291356a29b53722e4fddf2edb419fb84c5344deb364587403a800ad31"}
Nov 03 08:27:49.433572 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:49.433547    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5"
Nov 03 08:27:49.558602 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:49.558524    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7eb5a76-5765-468d-972b-08ce645c4fa8-kube-api-access-8jbsx\" (UniqueName: \"kubernetes.io/projected/c7eb5a76-5765-468d-972b-08ce645c4fa8-kube-api-access-8jbsx\") pod \"c7eb5a76-5765-468d-972b-08ce645c4fa8\" (UID: \"c7eb5a76-5765-468d-972b-08ce645c4fa8\") "
Nov 03 08:27:49.558789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:49.558611    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7eb5a76-5765-468d-972b-08ce645c4fa8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7eb5a76-5765-468d-972b-08ce645c4fa8-maas-api-openshift-ca\") pod \"c7eb5a76-5765-468d-972b-08ce645c4fa8\" (UID: \"c7eb5a76-5765-468d-972b-08ce645c4fa8\") "
Nov 03 08:27:49.558789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:49.558638    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7eb5a76-5765-468d-972b-08ce645c4fa8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7eb5a76-5765-468d-972b-08ce645c4fa8-maas-api-xks-ca\") pod \"c7eb5a76-5765-468d-972b-08ce645c4fa8\" (UID: \"c7eb5a76-5765-468d-972b-08ce645c4fa8\") "
Nov 03 08:27:49.559034 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:49.559010    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7eb5a76-5765-468d-972b-08ce645c4fa8-maas-api-openshift-ca" pod "c7eb5a76-5765-468d-972b-08ce645c4fa8" (UID: "c7eb5a76-5765-468d-972b-08ce645c4fa8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:27:49.560808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:49.560740    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7eb5a76-5765-468d-972b-08ce645c4fa8-kube-api-access-8jbsx" pod "c7eb5a76-5765-468d-972b-08ce645c4fa8" (UID: "c7eb5a76-5765-468d-972b-08ce645c4fa8"). InnerVolumeSpecName "kube-api-access-8jbsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:49.560808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:49.560749    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7eb5a76-5765-468d-972b-08ce645c4fa8-maas-api-xks-ca" pod "c7eb5a76-5765-468d-972b-08ce645c4fa8" (UID: "c7eb5a76-5765-468d-972b-08ce645c4fa8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:49.660299 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:49.660275    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jbsx\" (UniqueName: \"kubernetes.io/projected/c7eb5a76-5765-468d-972b-08ce645c4fa8-kube-api-access-8jbsx\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:49.660299 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:49.660299    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7eb5a76-5765-468d-972b-08ce645c4fa8-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:49.660458 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:49.660311    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7eb5a76-5765-468d-972b-08ce645c4fa8-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:50.296217 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:50.296154    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5"
Nov 03 08:27:50.296217 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:50.296189    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5" event={"ID":"c7eb5a76-5765-468d-972b-08ce645c4fa8","Type":"ContainerDied","Data":"33ca663a428d9c534f8455eb6307a920cca1cf1626df4d7e5d6d6b7b8c9484f8"}
Nov 03 08:27:50.296674 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:50.296235    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33ca663a428d9c534f8455eb6307a920cca1cf1626df4d7e5d6d6b7b8c9484f8"
Nov 03 08:27:50.463956 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:50.463935    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq"
Nov 03 08:27:50.582261 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:50.580499    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63e7b26a-4296-4429-9112-2e47f81773b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e7b26a-4296-4429-9112-2e47f81773b6-maas-api-xks-ca\") pod \"63e7b26a-4296-4429-9112-2e47f81773b6\" (UID: \"63e7b26a-4296-4429-9112-2e47f81773b6\") "
Nov 03 08:27:50.582261 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:50.580645    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63e7b26a-4296-4429-9112-2e47f81773b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e7b26a-4296-4429-9112-2e47f81773b6-maas-api-openshift-ca\") pod \"63e7b26a-4296-4429-9112-2e47f81773b6\" (UID: \"63e7b26a-4296-4429-9112-2e47f81773b6\") "
Nov 03 08:27:50.582261 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:50.581080    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e7b26a-4296-4429-9112-2e47f81773b6-maas-api-openshift-ca" pod "63e7b26a-4296-4429-9112-2e47f81773b6" (UID: "63e7b26a-4296-4429-9112-2e47f81773b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:27:50.582261 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:50.581289    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63e7b26a-4296-4429-9112-2e47f81773b6-kube-api-access-98x27\" (UniqueName: \"kubernetes.io/projected/63e7b26a-4296-4429-9112-2e47f81773b6-kube-api-access-98x27\") pod \"63e7b26a-4296-4429-9112-2e47f81773b6\" (UID: \"63e7b26a-4296-4429-9112-2e47f81773b6\") "
Nov 03 08:27:50.582261 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:50.581950    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63e7b26a-4296-4429-9112-2e47f81773b6-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:50.588373 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:50.588339    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63e7b26a-4296-4429-9112-2e47f81773b6-maas-api-xks-ca" pod "63e7b26a-4296-4429-9112-2e47f81773b6" (UID: "63e7b26a-4296-4429-9112-2e47f81773b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:50.590234 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:50.590189    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e7b26a-4296-4429-9112-2e47f81773b6-kube-api-access-98x27" pod "63e7b26a-4296-4429-9112-2e47f81773b6" (UID: "63e7b26a-4296-4429-9112-2e47f81773b6"). InnerVolumeSpecName "kube-api-access-98x27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:50.620410 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:50.620375    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"]
Nov 03 08:27:50.620643 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:50.620615    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b" podUID="8e96f256-aa30-4c3e-9d97-d829c96d13ac" containerName="istio-proxy" containerID="cri-o://3a6ebaa0a02caaeb1687c0a06c85877a161b816fa8de658d0e44407a72b8d44b" gracePeriod=30
Nov 03 08:27:50.682874 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:50.682845    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98x27\" (UniqueName: \"kubernetes.io/projected/63e7b26a-4296-4429-9112-2e47f81773b6-kube-api-access-98x27\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:50.682874 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:50.682869    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63e7b26a-4296-4429-9112-2e47f81773b6-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:51.105373 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.105330    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp"]
Nov 03 08:27:51.105980 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.105948    5070 cpu_manager.go:414] "containerMap: removing container" podUID="e569b99d-12a2-455b-9f87-45e229768318" containerName="revoke-keys"
Nov 03 08:27:51.105980 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.105980    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e569b99d-12a2-455b-9f87-45e229768318" containerName="revoke-keys"
Nov 03 08:27:51.106160 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.106032    5070 cpu_manager.go:414] "containerMap: removing container" podUID="63e7b26a-4296-4429-9112-2e47f81773b6" containerName="revoke-keys"
Nov 03 08:27:51.106160 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.106041    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e7b26a-4296-4429-9112-2e47f81773b6" containerName="revoke-keys"
Nov 03 08:27:51.106160 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.106058    5070 cpu_manager.go:414] "containerMap: removing container" podUID="c7eb5a76-5765-468d-972b-08ce645c4fa8" containerName="revoke-keys"
Nov 03 08:27:51.106160 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.106066    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7eb5a76-5765-468d-972b-08ce645c4fa8" containerName="revoke-keys"
Nov 03 08:27:51.106365 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.106232    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="e569b99d-12a2-455b-9f87-45e229768318" containerName="revoke-keys"
Nov 03 08:27:51.106365 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.106255    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e7b26a-4296-4429-9112-2e47f81773b6" containerName="revoke-keys"
Nov 03 08:27:51.106365 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.106267    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7eb5a76-5765-468d-972b-08ce645c4fa8" containerName="revoke-keys"
Nov 03 08:27:51.110033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.110010    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp"
Nov 03 08:27:51.120188 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.120161    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp"]
Nov 03 08:27:51.188983 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.188952    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca9deb0-1249-4558-87af-ccd2c0b0881b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp\" (UID: \"eca9deb0-1249-4558-87af-ccd2c0b0881b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp"
Nov 03 08:27:51.189141 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.189123    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca9deb0-1249-4558-87af-ccd2c0b0881b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp\" (UID: \"eca9deb0-1249-4558-87af-ccd2c0b0881b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp"
Nov 03 08:27:51.189194 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.189166    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcvpd\" (UniqueName: \"kubernetes.io/projected/eca9deb0-1249-4558-87af-ccd2c0b0881b-kube-api-access-lcvpd\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp\" (UID: \"eca9deb0-1249-4558-87af-ccd2c0b0881b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp"
Nov 03 08:27:51.290200 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.290172    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca9deb0-1249-4558-87af-ccd2c0b0881b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp\" (UID: \"eca9deb0-1249-4558-87af-ccd2c0b0881b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp"
Nov 03 08:27:51.290380 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.290264    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca9deb0-1249-4558-87af-ccd2c0b0881b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp\" (UID: \"eca9deb0-1249-4558-87af-ccd2c0b0881b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp"
Nov 03 08:27:51.290380 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.290283    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcvpd\" (UniqueName: \"kubernetes.io/projected/eca9deb0-1249-4558-87af-ccd2c0b0881b-kube-api-access-lcvpd\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp\" (UID: \"eca9deb0-1249-4558-87af-ccd2c0b0881b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp"
Nov 03 08:27:51.290915 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.290893    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca9deb0-1249-4558-87af-ccd2c0b0881b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp\" (UID: \"eca9deb0-1249-4558-87af-ccd2c0b0881b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp"
Nov 03 08:27:51.292650 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.292618    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca9deb0-1249-4558-87af-ccd2c0b0881b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp\" (UID: \"eca9deb0-1249-4558-87af-ccd2c0b0881b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp"
Nov 03 08:27:51.316960 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.316886    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcvpd\" (UniqueName: \"kubernetes.io/projected/eca9deb0-1249-4558-87af-ccd2c0b0881b-kube-api-access-lcvpd\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp\" (UID: \"eca9deb0-1249-4558-87af-ccd2c0b0881b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp"
Nov 03 08:27:51.318513 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.318494    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq"
Nov 03 08:27:51.318630 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.318530    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq" event={"ID":"63e7b26a-4296-4429-9112-2e47f81773b6","Type":"ContainerDied","Data":"1c4d0c562e3ac9bfe871953e9453cb58afa0711dcc1e29108ac80a6fd48ee4e5"}
Nov 03 08:27:51.318630 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.318598    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c4d0c562e3ac9bfe871953e9453cb58afa0711dcc1e29108ac80a6fd48ee4e5"
Nov 03 08:27:51.426701 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.426653    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp"
Nov 03 08:27:51.707492 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:51.707463    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp"]
Nov 03 08:27:51.708597 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:27:51.708573    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca9deb0_1249_4558_87af_ccd2c0b0881b.slice/crio-02ab09feaba9e64fae494ad2b36fda262015b484bfaced6b54a1dd36b73865dc WatchSource:0}: Error finding container 02ab09feaba9e64fae494ad2b36fda262015b484bfaced6b54a1dd36b73865dc: Status 404 returned error can't find the container with id 02ab09feaba9e64fae494ad2b36fda262015b484bfaced6b54a1dd36b73865dc
Nov 03 08:27:52.050799 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:27:52.050734    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:27:52.328234 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:52.328165    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp" event={"ID":"eca9deb0-1249-4558-87af-ccd2c0b0881b","Type":"ContainerStarted","Data":"f75911299efc04c7bdd5965f05f0fb5164e6d6c12246978597949704547c02b7"}
Nov 03 08:27:52.328234 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:52.328215    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp" event={"ID":"eca9deb0-1249-4558-87af-ccd2c0b0881b","Type":"ContainerStarted","Data":"02ab09feaba9e64fae494ad2b36fda262015b484bfaced6b54a1dd36b73865dc"}
Nov 03 08:27:52.345367 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:52.345326    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp" podStartSLOduration=1.345313781 podStartE2EDuration="1.345313781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:27:52.342425254 +0000 UTC m=+2070.456889187" watchObservedRunningTime="2026-11-03 08:27:52.345313781 +0000 UTC m=+2070.459777711"
Nov 03 08:27:53.071468 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.071431    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" containerID="cri-o://c140ff2083de5c92a6a58adde11782aaef9ac87a67ec52eb0697443528af6df8" gracePeriod=45
Nov 03 08:27:53.092458 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.092430    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main" containerID="cri-o://9c8106957f21cc7aa15ec81b8ce04508b85a41a78dabc8ff5e87c36cafc90bd0" gracePeriod=45
Nov 03 08:27:53.341830 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.340018    5070 generic.go:356] "Generic (PLEG): container finished" podID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerID="c140ff2083de5c92a6a58adde11782aaef9ac87a67ec52eb0697443528af6df8" exitCode=0
Nov 03 08:27:53.341830 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.340300    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" event={"ID":"b6603beb-ef9c-4cd4-96b2-286c95a37353","Type":"ContainerDied","Data":"c140ff2083de5c92a6a58adde11782aaef9ac87a67ec52eb0697443528af6df8"}
Nov 03 08:27:53.342484 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.342461    5070 generic.go:356] "Generic (PLEG): container finished" podID="eca9deb0-1249-4558-87af-ccd2c0b0881b" containerID="f75911299efc04c7bdd5965f05f0fb5164e6d6c12246978597949704547c02b7" exitCode=0
Nov 03 08:27:53.342567 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.342550    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp" event={"ID":"eca9deb0-1249-4558-87af-ccd2c0b0881b","Type":"ContainerDied","Data":"f75911299efc04c7bdd5965f05f0fb5164e6d6c12246978597949704547c02b7"}
Nov 03 08:27:53.344938 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.344910    5070 generic.go:356] "Generic (PLEG): container finished" podID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerID="9c8106957f21cc7aa15ec81b8ce04508b85a41a78dabc8ff5e87c36cafc90bd0" exitCode=0
Nov 03 08:27:53.345090 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.344992    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" event={"ID":"09a6416b-7395-4d6e-a6e5-45cf59873c6b","Type":"ContainerDied","Data":"9c8106957f21cc7aa15ec81b8ce04508b85a41a78dabc8ff5e87c36cafc90bd0"}
Nov 03 08:27:53.402792 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.390467    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:27:53.452636 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.452607    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:27:53.516329 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.516294    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a6416b-7395-4d6e-a6e5-45cf59873c6b-kube-api-access-464tx\" (UniqueName: \"kubernetes.io/projected/09a6416b-7395-4d6e-a6e5-45cf59873c6b-kube-api-access-464tx\") pod \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") "
Nov 03 08:27:53.516531 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.516386    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-dshm\") pod \"b6603beb-ef9c-4cd4-96b2-286c95a37353\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") "
Nov 03 08:27:53.516531 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.516425    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6603beb-ef9c-4cd4-96b2-286c95a37353-tls-certs\" (UniqueName: \"kubernetes.io/secret/b6603beb-ef9c-4cd4-96b2-286c95a37353-tls-certs\") pod \"b6603beb-ef9c-4cd4-96b2-286c95a37353\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") "
Nov 03 08:27:53.516531 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.516461    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09a6416b-7395-4d6e-a6e5-45cf59873c6b-tls-certs\" (UniqueName: \"kubernetes.io/secret/09a6416b-7395-4d6e-a6e5-45cf59873c6b-tls-certs\") pod \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") "
Nov 03 08:27:53.516531 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.516488    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-home\" (UniqueName: \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-home\") pod \"b6603beb-ef9c-4cd4-96b2-286c95a37353\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") "
Nov 03 08:27:53.516531 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.516521    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-tmp-dir\") pod \"b6603beb-ef9c-4cd4-96b2-286c95a37353\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") "
Nov 03 08:27:53.516825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.516569    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-dshm\") pod \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") "
Nov 03 08:27:53.516825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.516639    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6603beb-ef9c-4cd4-96b2-286c95a37353-kube-api-access-rs9fz\" (UniqueName: \"kubernetes.io/projected/b6603beb-ef9c-4cd4-96b2-286c95a37353-kube-api-access-rs9fz\") pod \"b6603beb-ef9c-4cd4-96b2-286c95a37353\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") "
Nov 03 08:27:53.516939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.516925    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-home\" (UniqueName: \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-home\") pod \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") "
Nov 03 08:27:53.516998 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.516960    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-tmp-dir\") pod \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") "
Nov 03 08:27:53.516998 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.516990    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-model-cache\") pod \"b6603beb-ef9c-4cd4-96b2-286c95a37353\" (UID: \"b6603beb-ef9c-4cd4-96b2-286c95a37353\") "
Nov 03 08:27:53.517108 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.517010    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-model-cache\") pod \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\" (UID: \"09a6416b-7395-4d6e-a6e5-45cf59873c6b\") "
Nov 03 08:27:53.517834 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.517804    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-model-cache" pod "09a6416b-7395-4d6e-a6e5-45cf59873c6b" (UID: "09a6416b-7395-4d6e-a6e5-45cf59873c6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:53.518848 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.518042    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-tmp-dir" pod "b6603beb-ef9c-4cd4-96b2-286c95a37353" (UID: "b6603beb-ef9c-4cd4-96b2-286c95a37353"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:53.518848 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.518379    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-tmp-dir" pod "09a6416b-7395-4d6e-a6e5-45cf59873c6b" (UID: "09a6416b-7395-4d6e-a6e5-45cf59873c6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:53.518848 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.518578    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-home" pod "09a6416b-7395-4d6e-a6e5-45cf59873c6b" (UID: "09a6416b-7395-4d6e-a6e5-45cf59873c6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:53.518848 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.518792    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-model-cache" pod "b6603beb-ef9c-4cd4-96b2-286c95a37353" (UID: "b6603beb-ef9c-4cd4-96b2-286c95a37353"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:53.519441 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.519417    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-home" pod "b6603beb-ef9c-4cd4-96b2-286c95a37353" (UID: "b6603beb-ef9c-4cd4-96b2-286c95a37353"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:53.521395 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.521369    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6603beb-ef9c-4cd4-96b2-286c95a37353-kube-api-access-rs9fz" pod "b6603beb-ef9c-4cd4-96b2-286c95a37353" (UID: "b6603beb-ef9c-4cd4-96b2-286c95a37353"). InnerVolumeSpecName "kube-api-access-rs9fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:53.521714 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.521690    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-dshm" pod "09a6416b-7395-4d6e-a6e5-45cf59873c6b" (UID: "09a6416b-7395-4d6e-a6e5-45cf59873c6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:53.521863 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.521842    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a6416b-7395-4d6e-a6e5-45cf59873c6b-kube-api-access-464tx" pod "09a6416b-7395-4d6e-a6e5-45cf59873c6b" (UID: "09a6416b-7395-4d6e-a6e5-45cf59873c6b"). InnerVolumeSpecName "kube-api-access-464tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:53.522560 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.522531    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-dshm" pod "b6603beb-ef9c-4cd4-96b2-286c95a37353" (UID: "b6603beb-ef9c-4cd4-96b2-286c95a37353"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:53.522929 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.522898    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6603beb-ef9c-4cd4-96b2-286c95a37353-tls-certs" pod "b6603beb-ef9c-4cd4-96b2-286c95a37353" (UID: "b6603beb-ef9c-4cd4-96b2-286c95a37353"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:53.523268 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.523241    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09a6416b-7395-4d6e-a6e5-45cf59873c6b-tls-certs" pod "09a6416b-7395-4d6e-a6e5-45cf59873c6b" (UID: "09a6416b-7395-4d6e-a6e5-45cf59873c6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:53.618787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.618739    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs9fz\" (UniqueName: \"kubernetes.io/projected/b6603beb-ef9c-4cd4-96b2-286c95a37353-kube-api-access-rs9fz\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:53.618787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.618783    5070 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-home\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:53.618787 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.618794    5070 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-tmp-dir\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:53.619033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.618803    5070 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-model-cache\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:53.619033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.618812    5070 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-model-cache\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:53.619033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.618821    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-464tx\" (UniqueName: \"kubernetes.io/projected/09a6416b-7395-4d6e-a6e5-45cf59873c6b-kube-api-access-464tx\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:53.619033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.618829    5070 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-dshm\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:53.619033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.618838    5070 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6603beb-ef9c-4cd4-96b2-286c95a37353-tls-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:53.619033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.618850    5070 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09a6416b-7395-4d6e-a6e5-45cf59873c6b-tls-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:53.619033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.618859    5070 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-home\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:53.619033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.618867    5070 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6603beb-ef9c-4cd4-96b2-286c95a37353-tmp-dir\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:53.619033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:53.618875    5070 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09a6416b-7395-4d6e-a6e5-45cf59873c6b-dshm\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:54.357035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:54.357009    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"
Nov 03 08:27:54.357449 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:54.357042    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" event={"ID":"b6603beb-ef9c-4cd4-96b2-286c95a37353","Type":"ContainerDied","Data":"252fb4e8a7ff5899c2cd9d9a3e39e64f17dc4ac0e78bb4ae436c307f57473512"}
Nov 03 08:27:54.357449 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:54.357092    5070 scope.go:122] "RemoveContainer" containerID="c140ff2083de5c92a6a58adde11782aaef9ac87a67ec52eb0697443528af6df8"
Nov 03 08:27:54.359039 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:54.359017    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"
Nov 03 08:27:54.359039 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:54.359026    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl" event={"ID":"09a6416b-7395-4d6e-a6e5-45cf59873c6b","Type":"ContainerDied","Data":"1e0b634b2da9c5155714a52ae7e6703445b8a86470fe5444cc24540f392ec887"}
Nov 03 08:27:54.365275 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:54.365247    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/ready\": context deadline exceeded"
Nov 03 08:27:54.371929 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:54.371909    5070 scope.go:122] "RemoveContainer" containerID="9c8106957f21cc7aa15ec81b8ce04508b85a41a78dabc8ff5e87c36cafc90bd0"
Nov 03 08:27:54.380613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:54.380590    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"]
Nov 03 08:27:54.407753 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:54.407705    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/distinct-w1-19bb0a-kserve-6cbfc66749-q2cl5"]
Nov 03 08:27:54.412741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:54.412713    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"]
Nov 03 08:27:54.416432 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:54.416407    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-19bb0a/simulator-w1-19bb0a-kserve-84b8688499-5nfnl"]
Nov 03 08:27:54.545665 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:54.545641    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp"
Nov 03 08:27:54.630598 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:54.630492    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eca9deb0-1249-4558-87af-ccd2c0b0881b-kube-api-access-lcvpd\" (UniqueName: \"kubernetes.io/projected/eca9deb0-1249-4558-87af-ccd2c0b0881b-kube-api-access-lcvpd\") pod \"eca9deb0-1249-4558-87af-ccd2c0b0881b\" (UID: \"eca9deb0-1249-4558-87af-ccd2c0b0881b\") "
Nov 03 08:27:54.630819 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:54.630615    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eca9deb0-1249-4558-87af-ccd2c0b0881b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca9deb0-1249-4558-87af-ccd2c0b0881b-maas-api-xks-ca\") pod \"eca9deb0-1249-4558-87af-ccd2c0b0881b\" (UID: \"eca9deb0-1249-4558-87af-ccd2c0b0881b\") "
Nov 03 08:27:54.630922 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:54.630818    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eca9deb0-1249-4558-87af-ccd2c0b0881b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca9deb0-1249-4558-87af-ccd2c0b0881b-maas-api-openshift-ca\") pod \"eca9deb0-1249-4558-87af-ccd2c0b0881b\" (UID: \"eca9deb0-1249-4558-87af-ccd2c0b0881b\") "
Nov 03 08:27:54.631195 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:54.631154    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca9deb0-1249-4558-87af-ccd2c0b0881b-maas-api-openshift-ca" pod "eca9deb0-1249-4558-87af-ccd2c0b0881b" (UID: "eca9deb0-1249-4558-87af-ccd2c0b0881b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:27:54.631330 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:54.631308    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eca9deb0-1249-4558-87af-ccd2c0b0881b-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:54.633182 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:54.633149    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca9deb0-1249-4558-87af-ccd2c0b0881b-kube-api-access-lcvpd" pod "eca9deb0-1249-4558-87af-ccd2c0b0881b" (UID: "eca9deb0-1249-4558-87af-ccd2c0b0881b"). InnerVolumeSpecName "kube-api-access-lcvpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:54.633479 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:54.633454    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eca9deb0-1249-4558-87af-ccd2c0b0881b-maas-api-xks-ca" pod "eca9deb0-1249-4558-87af-ccd2c0b0881b" (UID: "eca9deb0-1249-4558-87af-ccd2c0b0881b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:27:54.752784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:54.748821    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcvpd\" (UniqueName: \"kubernetes.io/projected/eca9deb0-1249-4558-87af-ccd2c0b0881b-kube-api-access-lcvpd\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:54.752784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:54.748869    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eca9deb0-1249-4558-87af-ccd2c0b0881b-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:55.367826 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:55.367788    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp"
Nov 03 08:27:55.367826 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:55.367798    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp" event={"ID":"eca9deb0-1249-4558-87af-ccd2c0b0881b","Type":"ContainerDied","Data":"02ab09feaba9e64fae494ad2b36fda262015b484bfaced6b54a1dd36b73865dc"}
Nov 03 08:27:55.368304 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:55.367855    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02ab09feaba9e64fae494ad2b36fda262015b484bfaced6b54a1dd36b73865dc"
Nov 03 08:27:55.904004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:55.903954    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:27:56.025906 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.025879    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" path="/var/lib/kubelet/pods/09a6416b-7395-4d6e-a6e5-45cf59873c6b/volumes"
Nov 03 08:27:56.026323 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.026308    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" path="/var/lib/kubelet/pods/b6603beb-ef9c-4cd4-96b2-286c95a37353/volumes"
Nov 03 08:27:56.066718 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.066680    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istiod-ca-cert\") pod \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") "
Nov 03 08:27:56.067054 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.066725    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-token\" (UniqueName: \"kubernetes.io/projected/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-token\") pod \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") "
Nov 03 08:27:56.067054 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.066800    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-envoy\") pod \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") "
Nov 03 08:27:56.067054 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.066966    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-data\") pod \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") "
Nov 03 08:27:56.067054 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.067012    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-workload-socket\") pod \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") "
Nov 03 08:27:56.067310 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.067079    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-workload-certs\") pod \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") "
Nov 03 08:27:56.067310 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.067103    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istiod-ca-cert" pod "8e96f256-aa30-4c3e-9d97-d829c96d13ac" (UID: "8e96f256-aa30-4c3e-9d97-d829c96d13ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:27:56.067310 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.067121    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-podinfo\") pod \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") "
Nov 03 08:27:56.067310 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.067268    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-data" pod "8e96f256-aa30-4c3e-9d97-d829c96d13ac" (UID: "8e96f256-aa30-4c3e-9d97-d829c96d13ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:56.067553 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.067319    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-workload-socket" pod "8e96f256-aa30-4c3e-9d97-d829c96d13ac" (UID: "8e96f256-aa30-4c3e-9d97-d829c96d13ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:56.067553 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.067273    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e96f256-aa30-4c3e-9d97-d829c96d13ac-kube-api-access-stb4t\" (UniqueName: \"kubernetes.io/projected/8e96f256-aa30-4c3e-9d97-d829c96d13ac-kube-api-access-stb4t\") pod \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") "
Nov 03 08:27:56.067553 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.067419    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-credential-socket\") pod \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\" (UID: \"8e96f256-aa30-4c3e-9d97-d829c96d13ac\") "
Nov 03 08:27:56.067553 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.067443    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-workload-certs" pod "8e96f256-aa30-4c3e-9d97-d829c96d13ac" (UID: "8e96f256-aa30-4c3e-9d97-d829c96d13ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:56.067820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.067635    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-credential-socket" pod "8e96f256-aa30-4c3e-9d97-d829c96d13ac" (UID: "8e96f256-aa30-4c3e-9d97-d829c96d13ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:56.068022 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.068002    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:56.068099 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.068022    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:56.068099 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.068033    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:56.068099 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.068041    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:56.068099 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.068051    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:56.069243 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.069220    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-podinfo" pod "8e96f256-aa30-4c3e-9d97-d829c96d13ac" (UID: "8e96f256-aa30-4c3e-9d97-d829c96d13ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:27:56.069443 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.069424    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-token" pod "8e96f256-aa30-4c3e-9d97-d829c96d13ac" (UID: "8e96f256-aa30-4c3e-9d97-d829c96d13ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:56.069526 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.069505    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-envoy" pod "8e96f256-aa30-4c3e-9d97-d829c96d13ac" (UID: "8e96f256-aa30-4c3e-9d97-d829c96d13ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:27:56.069639 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.069621    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e96f256-aa30-4c3e-9d97-d829c96d13ac-kube-api-access-stb4t" pod "8e96f256-aa30-4c3e-9d97-d829c96d13ac" (UID: "8e96f256-aa30-4c3e-9d97-d829c96d13ac"). InnerVolumeSpecName "kube-api-access-stb4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:27:56.169211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.169154    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:56.169211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.169179    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stb4t\" (UniqueName: \"kubernetes.io/projected/8e96f256-aa30-4c3e-9d97-d829c96d13ac-kube-api-access-stb4t\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:56.169211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.169195    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:56.169211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.169208    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e96f256-aa30-4c3e-9d97-d829c96d13ac-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:27:56.380469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.380440    5070 generic.go:356] "Generic (PLEG): container finished" podID="8e96f256-aa30-4c3e-9d97-d829c96d13ac" containerID="3a6ebaa0a02caaeb1687c0a06c85877a161b816fa8de658d0e44407a72b8d44b" exitCode=0
Nov 03 08:27:56.380846 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.380492    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"
Nov 03 08:27:56.380846 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.380519    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b" event={"ID":"8e96f256-aa30-4c3e-9d97-d829c96d13ac","Type":"ContainerDied","Data":"3a6ebaa0a02caaeb1687c0a06c85877a161b816fa8de658d0e44407a72b8d44b"}
Nov 03 08:27:56.380846 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.380577    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b" event={"ID":"8e96f256-aa30-4c3e-9d97-d829c96d13ac","Type":"ContainerDied","Data":"8f13227603bae4ee40deb029657293da112550ec4b0211d1830220a4ef39acf8"}
Nov 03 08:27:56.380846 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.380600    5070 scope.go:122] "RemoveContainer" containerID="3a6ebaa0a02caaeb1687c0a06c85877a161b816fa8de658d0e44407a72b8d44b"
Nov 03 08:27:56.392812 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.392796    5070 scope.go:122] "RemoveContainer" containerID="f05d3136eb9be54b7c1b46679e84dab6c1c0ede5c0327ab9abf0ed279871202e"
Nov 03 08:27:56.404003 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.403979    5070 scope.go:122] "RemoveContainer" containerID="3a6ebaa0a02caaeb1687c0a06c85877a161b816fa8de658d0e44407a72b8d44b"
Nov 03 08:27:56.404287 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:27:56.404261    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a6ebaa0a02caaeb1687c0a06c85877a161b816fa8de658d0e44407a72b8d44b\": container with ID starting with 3a6ebaa0a02caaeb1687c0a06c85877a161b816fa8de658d0e44407a72b8d44b not found: ID does not exist" containerID="3a6ebaa0a02caaeb1687c0a06c85877a161b816fa8de658d0e44407a72b8d44b"
Nov 03 08:27:56.404348 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.404289    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a6ebaa0a02caaeb1687c0a06c85877a161b816fa8de658d0e44407a72b8d44b"} err="failed to get container status \"3a6ebaa0a02caaeb1687c0a06c85877a161b816fa8de658d0e44407a72b8d44b\": rpc error: code = NotFound desc = could not find container \"3a6ebaa0a02caaeb1687c0a06c85877a161b816fa8de658d0e44407a72b8d44b\": container with ID starting with 3a6ebaa0a02caaeb1687c0a06c85877a161b816fa8de658d0e44407a72b8d44b not found: ID does not exist"
Nov 03 08:27:56.404348 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.404306    5070 scope.go:122] "RemoveContainer" containerID="f05d3136eb9be54b7c1b46679e84dab6c1c0ede5c0327ab9abf0ed279871202e"
Nov 03 08:27:56.404444 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.404400    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"]
Nov 03 08:27:56.404545 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:27:56.404525    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f05d3136eb9be54b7c1b46679e84dab6c1c0ede5c0327ab9abf0ed279871202e\": container with ID starting with f05d3136eb9be54b7c1b46679e84dab6c1c0ede5c0327ab9abf0ed279871202e not found: ID does not exist" containerID="f05d3136eb9be54b7c1b46679e84dab6c1c0ede5c0327ab9abf0ed279871202e"
Nov 03 08:27:56.404592 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.404552    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f05d3136eb9be54b7c1b46679e84dab6c1c0ede5c0327ab9abf0ed279871202e"} err="failed to get container status \"f05d3136eb9be54b7c1b46679e84dab6c1c0ede5c0327ab9abf0ed279871202e\": rpc error: code = NotFound desc = could not find container \"f05d3136eb9be54b7c1b46679e84dab6c1c0ede5c0327ab9abf0ed279871202e\": container with ID starting with f05d3136eb9be54b7c1b46679e84dab6c1c0ede5c0327ab9abf0ed279871202e not found: ID does not exist"
Nov 03 08:27:56.408097 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:56.408078    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-109070-openshift-default-7d8558c575-6mk9b"]
Nov 03 08:27:58.037796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:58.037739    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e96f256-aa30-4c3e-9d97-d829c96d13ac" path="/var/lib/kubelet/pods/8e96f256-aa30-4c3e-9d97-d829c96d13ac/volumes"
Nov 03 08:27:58.189283 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:58.189240    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg" podUID="a2bafc12-e2f3-46ee-8f98-01e46c2c2787" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:27:58.336693 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:58.336600    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp"]
Nov 03 08:27:58.341102 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:58.341071    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-19bb0a-qkzgp"]
Nov 03 08:27:59.739658 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:59.739623    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq"]
Nov 03 08:27:59.743905 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:27:59.743878    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0fc79b-f54qq"]
Nov 03 08:28:00.025147 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:00.025073    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63e7b26a-4296-4429-9112-2e47f81773b6" path="/var/lib/kubelet/pods/63e7b26a-4296-4429-9112-2e47f81773b6/volumes"
Nov 03 08:28:00.025494 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:00.025479    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4d646a8-437f-44a9-861a-a704958b3e12" path="/var/lib/kubelet/pods/c4d646a8-437f-44a9-861a-a704958b3e12/volumes"
Nov 03 08:28:02.637681 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:02.637648    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"]
Nov 03 08:28:02.638105 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:02.637864    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g" podUID="9f09e1af-345e-4489-806a-b3aa7b8cfbd8" containerName="discovery" containerID="cri-o://cd1dd43f1a57f2fc7b32734b70073370e497c19ac9de054664734a80bb610267" gracePeriod=30
Nov 03 08:28:02.949166 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:02.949089    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:28:03.052600 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.052570    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-token\" (UniqueName: \"kubernetes.io/projected/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-token\") pod \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") "
Nov 03 08:28:03.052823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.052659    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-kube-api-access-zjnn4\" (UniqueName: \"kubernetes.io/projected/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-kube-api-access-zjnn4\") pod \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") "
Nov 03 08:28:03.052823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.052679    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-csr-dns-cert\") pod \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") "
Nov 03 08:28:03.052823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.052714    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-local-certs\") pod \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") "
Nov 03 08:28:03.053022 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.052828    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-cacerts\" (UniqueName: \"kubernetes.io/secret/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-cacerts\") pod \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") "
Nov 03 08:28:03.053022 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.052851    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-csr-ca-configmap\") pod \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") "
Nov 03 08:28:03.053022 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.052869    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-kubeconfig\") pod \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\" (UID: \"9f09e1af-345e-4489-806a-b3aa7b8cfbd8\") "
Nov 03 08:28:03.053582 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.053540    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-csr-ca-configmap" pod "9f09e1af-345e-4489-806a-b3aa7b8cfbd8" (UID: "9f09e1af-345e-4489-806a-b3aa7b8cfbd8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:03.055593 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.055566    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-token" pod "9f09e1af-345e-4489-806a-b3aa7b8cfbd8" (UID: "9f09e1af-345e-4489-806a-b3aa7b8cfbd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:03.055722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.055612    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-kube-api-access-zjnn4" pod "9f09e1af-345e-4489-806a-b3aa7b8cfbd8" (UID: "9f09e1af-345e-4489-806a-b3aa7b8cfbd8"). InnerVolumeSpecName "kube-api-access-zjnn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:03.075775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.072965    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-local-certs" pod "9f09e1af-345e-4489-806a-b3aa7b8cfbd8" (UID: "9f09e1af-345e-4489-806a-b3aa7b8cfbd8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:03.075775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.073948    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-cacerts" pod "9f09e1af-345e-4489-806a-b3aa7b8cfbd8" (UID: "9f09e1af-345e-4489-806a-b3aa7b8cfbd8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:03.081035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.080778    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-kubeconfig" pod "9f09e1af-345e-4489-806a-b3aa7b8cfbd8" (UID: "9f09e1af-345e-4489-806a-b3aa7b8cfbd8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:03.081035 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.080896    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-csr-dns-cert" pod "9f09e1af-345e-4489-806a-b3aa7b8cfbd8" (UID: "9f09e1af-345e-4489-806a-b3aa7b8cfbd8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:03.154063 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.154035    5070 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-cacerts\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:03.154063 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.154060    5070 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-csr-ca-configmap\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:03.154063 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.154069    5070 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-kubeconfig\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:03.154311 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.154081    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:03.154311 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.154091    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjnn4\" (UniqueName: \"kubernetes.io/projected/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-kube-api-access-zjnn4\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:03.154311 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.154100    5070 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-istio-csr-dns-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:03.154311 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.154109    5070 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f09e1af-345e-4489-806a-b3aa7b8cfbd8-local-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:03.411183 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.411146    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"]
Nov 03 08:28:03.411371 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.411350    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5" podUID="d1a356ec-fba8-45e1-87ba-13950f3e1fcb" containerName="istio-proxy" containerID="cri-o://17b6b191b3196f755413f0edc86314530820822b4d582d2d02ab4c52da8baece" gracePeriod=30
Nov 03 08:28:03.444505 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.444469    5070 generic.go:356] "Generic (PLEG): container finished" podID="9f09e1af-345e-4489-806a-b3aa7b8cfbd8" containerID="cd1dd43f1a57f2fc7b32734b70073370e497c19ac9de054664734a80bb610267" exitCode=0
Nov 03 08:28:03.444708 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.444529    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"
Nov 03 08:28:03.444708 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.444555    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g" event={"ID":"9f09e1af-345e-4489-806a-b3aa7b8cfbd8","Type":"ContainerDied","Data":"cd1dd43f1a57f2fc7b32734b70073370e497c19ac9de054664734a80bb610267"}
Nov 03 08:28:03.444708 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.444612    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g" event={"ID":"9f09e1af-345e-4489-806a-b3aa7b8cfbd8","Type":"ContainerDied","Data":"4b84c50a558fe47f9e6e6da0b29069f6ffd515bce92014548b3519413af23082"}
Nov 03 08:28:03.444708 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.444639    5070 scope.go:122] "RemoveContainer" containerID="cd1dd43f1a57f2fc7b32734b70073370e497c19ac9de054664734a80bb610267"
Nov 03 08:28:03.489779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.489743    5070 scope.go:122] "RemoveContainer" containerID="cd1dd43f1a57f2fc7b32734b70073370e497c19ac9de054664734a80bb610267"
Nov 03 08:28:03.490302 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:28:03.490139    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd1dd43f1a57f2fc7b32734b70073370e497c19ac9de054664734a80bb610267\": container with ID starting with cd1dd43f1a57f2fc7b32734b70073370e497c19ac9de054664734a80bb610267 not found: ID does not exist" containerID="cd1dd43f1a57f2fc7b32734b70073370e497c19ac9de054664734a80bb610267"
Nov 03 08:28:03.490302 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.490182    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd1dd43f1a57f2fc7b32734b70073370e497c19ac9de054664734a80bb610267"} err="failed to get container status \"cd1dd43f1a57f2fc7b32734b70073370e497c19ac9de054664734a80bb610267\": rpc error: code = NotFound desc = could not find container \"cd1dd43f1a57f2fc7b32734b70073370e497c19ac9de054664734a80bb610267\": container with ID starting with cd1dd43f1a57f2fc7b32734b70073370e497c19ac9de054664734a80bb610267 not found: ID does not exist"
Nov 03 08:28:03.498803 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.498514    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"]
Nov 03 08:28:03.502201 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:03.502180    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zmb7g"]
Nov 03 08:28:04.025423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:04.025390    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f09e1af-345e-4489-806a-b3aa7b8cfbd8" path="/var/lib/kubelet/pods/9f09e1af-345e-4489-806a-b3aa7b8cfbd8/volumes"
Nov 03 08:28:05.585881 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:05.583869    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp"]
Nov 03 08:28:05.605622 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:05.605586    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-178e68-8b2wp"]
Nov 03 08:28:05.785276 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:05.785246    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:28:06.025911 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:06.025875    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eca9deb0-1249-4558-87af-ccd2c0b0881b" path="/var/lib/kubelet/pods/eca9deb0-1249-4558-87af-ccd2c0b0881b/volumes"
Nov 03 08:28:06.897796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:06.896808    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"]
Nov 03 08:28:06.897796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:06.897006    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt" podUID="f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" containerName="istio-proxy" containerID="cri-o://9c17e76f83d02982413d20c13e208ed6683938d3c14d74b497e6d9aa053b41ff" gracePeriod=30
Nov 03 08:28:08.011982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.011946    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"]
Nov 03 08:28:08.012469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.012204    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg" podUID="79b2069b-a7c8-45eb-9739-9bf3a1fec044" containerName="istio-proxy" containerID="cri-o://772ca4ffdb2c297a843bbf7b384c2a89d75821e380e310aeee6b4907ffe91c21" gracePeriod=30
Nov 03 08:28:08.187419 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.187376    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg" podUID="a2bafc12-e2f3-46ee-8f98-01e46c2c2787" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:28:08.488862 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.488827    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t"]
Nov 03 08:28:08.489361 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.489346    5070 cpu_manager.go:414] "containerMap: removing container" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main"
Nov 03 08:28:08.489431 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.489363    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main"
Nov 03 08:28:08.489431 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.489377    5070 cpu_manager.go:414] "containerMap: removing container" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main"
Nov 03 08:28:08.489431 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.489384    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main"
Nov 03 08:28:08.489431 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.489400    5070 cpu_manager.go:414] "containerMap: removing container" podUID="eca9deb0-1249-4558-87af-ccd2c0b0881b" containerName="revoke-keys"
Nov 03 08:28:08.489431 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.489407    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eca9deb0-1249-4558-87af-ccd2c0b0881b" containerName="revoke-keys"
Nov 03 08:28:08.489431 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.489430    5070 cpu_manager.go:414] "containerMap: removing container" podUID="8e96f256-aa30-4c3e-9d97-d829c96d13ac" containerName="istio-proxy"
Nov 03 08:28:08.489707 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.489435    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e96f256-aa30-4c3e-9d97-d829c96d13ac" containerName="istio-proxy"
Nov 03 08:28:08.489707 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.489450    5070 cpu_manager.go:414] "containerMap: removing container" podUID="9f09e1af-345e-4489-806a-b3aa7b8cfbd8" containerName="discovery"
Nov 03 08:28:08.489707 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.489455    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f09e1af-345e-4489-806a-b3aa7b8cfbd8" containerName="discovery"
Nov 03 08:28:08.489707 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.489467    5070 cpu_manager.go:414] "containerMap: removing container" podUID="8e96f256-aa30-4c3e-9d97-d829c96d13ac" containerName="istio-proxy"
Nov 03 08:28:08.489707 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.489471    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e96f256-aa30-4c3e-9d97-d829c96d13ac" containerName="istio-proxy"
Nov 03 08:28:08.489707 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.489550    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e96f256-aa30-4c3e-9d97-d829c96d13ac" containerName="istio-proxy"
Nov 03 08:28:08.489707 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.489561    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f09e1af-345e-4489-806a-b3aa7b8cfbd8" containerName="discovery"
Nov 03 08:28:08.489707 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.489567    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e96f256-aa30-4c3e-9d97-d829c96d13ac" containerName="istio-proxy"
Nov 03 08:28:08.489707 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.489575    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6603beb-ef9c-4cd4-96b2-286c95a37353" containerName="main"
Nov 03 08:28:08.489707 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.489581    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a6416b-7395-4d6e-a6e5-45cf59873c6b" containerName="main"
Nov 03 08:28:08.489707 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.489587    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="eca9deb0-1249-4558-87af-ccd2c0b0881b" containerName="revoke-keys"
Nov 03 08:28:08.497934 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.493032    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t"
Nov 03 08:28:08.499079 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.499045    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:28:08.499799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.499776    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:28:08.501002 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.500975    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t"]
Nov 03 08:28:08.522096 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.521969    5070 generic.go:356] "Generic (PLEG): container finished" podID="a2bafc12-e2f3-46ee-8f98-01e46c2c2787" containerID="9a349cac83afa17a6fd7dcd2183a4c8615aecc6cdb9c5ff2a4897155383bcb1f" exitCode=0
Nov 03 08:28:08.522096 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.522029    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg" event={"ID":"a2bafc12-e2f3-46ee-8f98-01e46c2c2787","Type":"ContainerDied","Data":"9a349cac83afa17a6fd7dcd2183a4c8615aecc6cdb9c5ff2a4897155383bcb1f"}
Nov 03 08:28:08.539722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.539575    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:28:08.539722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.539672    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:28:08.545406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.545264    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w46nr\" (UniqueName: \"kubernetes.io/projected/961562d5-1288-4dbc-9723-451021080e30-kube-api-access-w46nr\") pod \"maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t\" (UID: \"961562d5-1288-4dbc-9723-451021080e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t"
Nov 03 08:28:08.545406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.545352    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/961562d5-1288-4dbc-9723-451021080e30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t\" (UID: \"961562d5-1288-4dbc-9723-451021080e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t"
Nov 03 08:28:08.545620 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.545510    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/961562d5-1288-4dbc-9723-451021080e30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t\" (UID: \"961562d5-1288-4dbc-9723-451021080e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t"
Nov 03 08:28:08.647558 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.646715    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/961562d5-1288-4dbc-9723-451021080e30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t\" (UID: \"961562d5-1288-4dbc-9723-451021080e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t"
Nov 03 08:28:08.647558 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.646841    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w46nr\" (UniqueName: \"kubernetes.io/projected/961562d5-1288-4dbc-9723-451021080e30-kube-api-access-w46nr\") pod \"maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t\" (UID: \"961562d5-1288-4dbc-9723-451021080e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t"
Nov 03 08:28:08.647558 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.646932    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/961562d5-1288-4dbc-9723-451021080e30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t\" (UID: \"961562d5-1288-4dbc-9723-451021080e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t"
Nov 03 08:28:08.648576 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.648539    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/961562d5-1288-4dbc-9723-451021080e30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t\" (UID: \"961562d5-1288-4dbc-9723-451021080e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t"
Nov 03 08:28:08.655560 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.655506    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/961562d5-1288-4dbc-9723-451021080e30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t\" (UID: \"961562d5-1288-4dbc-9723-451021080e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t"
Nov 03 08:28:08.681114 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.681082    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w46nr\" (UniqueName: \"kubernetes.io/projected/961562d5-1288-4dbc-9723-451021080e30-kube-api-access-w46nr\") pod \"maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t\" (UID: \"961562d5-1288-4dbc-9723-451021080e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t"
Nov 03 08:28:08.718234 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.717938    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg"
Nov 03 08:28:08.749028 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.748937    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2bafc12-e2f3-46ee-8f98-01e46c2c2787-kube-api-access-7n7sr\" (UniqueName: \"kubernetes.io/projected/a2bafc12-e2f3-46ee-8f98-01e46c2c2787-kube-api-access-7n7sr\") pod \"a2bafc12-e2f3-46ee-8f98-01e46c2c2787\" (UID: \"a2bafc12-e2f3-46ee-8f98-01e46c2c2787\") "
Nov 03 08:28:08.749028 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.748980    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2bafc12-e2f3-46ee-8f98-01e46c2c2787-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2bafc12-e2f3-46ee-8f98-01e46c2c2787-plugins-config-volume\") pod \"a2bafc12-e2f3-46ee-8f98-01e46c2c2787\" (UID: \"a2bafc12-e2f3-46ee-8f98-01e46c2c2787\") "
Nov 03 08:28:08.749466 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.749436    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2bafc12-e2f3-46ee-8f98-01e46c2c2787-plugins-config-volume" pod "a2bafc12-e2f3-46ee-8f98-01e46c2c2787" (UID: "a2bafc12-e2f3-46ee-8f98-01e46c2c2787"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:08.751860 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.751650    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2bafc12-e2f3-46ee-8f98-01e46c2c2787-kube-api-access-7n7sr" pod "a2bafc12-e2f3-46ee-8f98-01e46c2c2787" (UID: "a2bafc12-e2f3-46ee-8f98-01e46c2c2787"). InnerVolumeSpecName "kube-api-access-7n7sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:08.764541 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.764517    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:28:08.839091 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.839063    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t"
Nov 03 08:28:08.850679 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.850654    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-kube-api-access-ljjfs\" (UniqueName: \"kubernetes.io/projected/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-kube-api-access-ljjfs\") pod \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") "
Nov 03 08:28:08.850877 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.850715    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-token\" (UniqueName: \"kubernetes.io/projected/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-token\") pod \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") "
Nov 03 08:28:08.850877 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.850744    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-podinfo\") pod \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") "
Nov 03 08:28:08.850877 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.850785    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-credential-socket\") pod \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") "
Nov 03 08:28:08.850877 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.850869    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istiod-ca-cert\") pod \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") "
Nov 03 08:28:08.851115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.850904    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-workload-certs\") pod \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") "
Nov 03 08:28:08.851115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.850938    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-envoy\") pod \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") "
Nov 03 08:28:08.851115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.850988    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-workload-socket\") pod \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") "
Nov 03 08:28:08.851115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.851032    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-data\") pod \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\" (UID: \"d1a356ec-fba8-45e1-87ba-13950f3e1fcb\") "
Nov 03 08:28:08.851436 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.851357    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-workload-certs" pod "d1a356ec-fba8-45e1-87ba-13950f3e1fcb" (UID: "d1a356ec-fba8-45e1-87ba-13950f3e1fcb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:08.851436 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.851399    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-credential-socket" pod "d1a356ec-fba8-45e1-87ba-13950f3e1fcb" (UID: "d1a356ec-fba8-45e1-87ba-13950f3e1fcb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:08.851991 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.851626    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:08.851991 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.851651    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n7sr\" (UniqueName: \"kubernetes.io/projected/a2bafc12-e2f3-46ee-8f98-01e46c2c2787-kube-api-access-7n7sr\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:08.851991 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.851666    5070 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2bafc12-e2f3-46ee-8f98-01e46c2c2787-plugins-config-volume\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:08.851991 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.851677    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:08.851991 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.851695    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istiod-ca-cert" pod "d1a356ec-fba8-45e1-87ba-13950f3e1fcb" (UID: "d1a356ec-fba8-45e1-87ba-13950f3e1fcb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:08.851991 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.851958    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-workload-socket" pod "d1a356ec-fba8-45e1-87ba-13950f3e1fcb" (UID: "d1a356ec-fba8-45e1-87ba-13950f3e1fcb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:08.852389 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.852101    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-data" pod "d1a356ec-fba8-45e1-87ba-13950f3e1fcb" (UID: "d1a356ec-fba8-45e1-87ba-13950f3e1fcb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:08.853569 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.853547    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-kube-api-access-ljjfs" pod "d1a356ec-fba8-45e1-87ba-13950f3e1fcb" (UID: "d1a356ec-fba8-45e1-87ba-13950f3e1fcb"). InnerVolumeSpecName "kube-api-access-ljjfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:08.853838 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.853817    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-envoy" pod "d1a356ec-fba8-45e1-87ba-13950f3e1fcb" (UID: "d1a356ec-fba8-45e1-87ba-13950f3e1fcb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:08.853919 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.853877    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-token" pod "d1a356ec-fba8-45e1-87ba-13950f3e1fcb" (UID: "d1a356ec-fba8-45e1-87ba-13950f3e1fcb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:08.854402 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.854383    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-podinfo" pod "d1a356ec-fba8-45e1-87ba-13950f3e1fcb" (UID: "d1a356ec-fba8-45e1-87ba-13950f3e1fcb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:28:08.952708 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.952668    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:08.952708 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.952704    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:08.952941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.952718    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:08.952941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.952732    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:08.952941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.952744    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljjfs\" (UniqueName: \"kubernetes.io/projected/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-kube-api-access-ljjfs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:08.952941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.952770    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:08.952941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:08.952782    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a356ec-fba8-45e1-87ba-13950f3e1fcb-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:09.120097 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:28:09.120061    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961562d5_1288_4dbc_9723_451021080e30.slice/crio-a38b83a441097e99ea392de447a8d9a4023bce5e06de5fe1b6ce84ed53b27717 WatchSource:0}: Error finding container a38b83a441097e99ea392de447a8d9a4023bce5e06de5fe1b6ce84ed53b27717: Status 404 returned error can't find the container with id a38b83a441097e99ea392de447a8d9a4023bce5e06de5fe1b6ce84ed53b27717
Nov 03 08:28:09.121418 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:09.121388    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t"]
Nov 03 08:28:09.532819 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:09.532791    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t" event={"ID":"961562d5-1288-4dbc-9723-451021080e30","Type":"ContainerStarted","Data":"0999cd513af3becb230dfc5833512911040d67852d69a7e4fba157333c7ab733"}
Nov 03 08:28:09.532984 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:09.532841    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t" event={"ID":"961562d5-1288-4dbc-9723-451021080e30","Type":"ContainerStarted","Data":"a38b83a441097e99ea392de447a8d9a4023bce5e06de5fe1b6ce84ed53b27717"}
Nov 03 08:28:09.534284 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:09.534262    5070 generic.go:356] "Generic (PLEG): container finished" podID="d1a356ec-fba8-45e1-87ba-13950f3e1fcb" containerID="17b6b191b3196f755413f0edc86314530820822b4d582d2d02ab4c52da8baece" exitCode=0
Nov 03 08:28:09.534463 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:09.534290    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5" event={"ID":"d1a356ec-fba8-45e1-87ba-13950f3e1fcb","Type":"ContainerDied","Data":"17b6b191b3196f755413f0edc86314530820822b4d582d2d02ab4c52da8baece"}
Nov 03 08:28:09.534463 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:09.534312    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"
Nov 03 08:28:09.534463 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:09.534333    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5" event={"ID":"d1a356ec-fba8-45e1-87ba-13950f3e1fcb","Type":"ContainerDied","Data":"ffd2bad394dae91aea11bdff6265f773dee642327324ca96bc1f77361e713a5c"}
Nov 03 08:28:09.534463 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:09.534356    5070 scope.go:122] "RemoveContainer" containerID="17b6b191b3196f755413f0edc86314530820822b4d582d2d02ab4c52da8baece"
Nov 03 08:28:09.535865 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:09.535846    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg"
Nov 03 08:28:09.535958 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:09.535880    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg" event={"ID":"a2bafc12-e2f3-46ee-8f98-01e46c2c2787","Type":"ContainerDied","Data":"cd2d5b6a6020f6fb19771efe29f05a5c406c6d58222f86f6349bab46ccab1589"}
Nov 03 08:28:09.546530 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:09.546512    5070 scope.go:122] "RemoveContainer" containerID="17b6b191b3196f755413f0edc86314530820822b4d582d2d02ab4c52da8baece"
Nov 03 08:28:09.546884 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:28:09.546864    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17b6b191b3196f755413f0edc86314530820822b4d582d2d02ab4c52da8baece\": container with ID starting with 17b6b191b3196f755413f0edc86314530820822b4d582d2d02ab4c52da8baece not found: ID does not exist" containerID="17b6b191b3196f755413f0edc86314530820822b4d582d2d02ab4c52da8baece"
Nov 03 08:28:09.546994 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:09.546890    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b6b191b3196f755413f0edc86314530820822b4d582d2d02ab4c52da8baece"} err="failed to get container status \"17b6b191b3196f755413f0edc86314530820822b4d582d2d02ab4c52da8baece\": rpc error: code = NotFound desc = could not find container \"17b6b191b3196f755413f0edc86314530820822b4d582d2d02ab4c52da8baece\": container with ID starting with 17b6b191b3196f755413f0edc86314530820822b4d582d2d02ab4c52da8baece not found: ID does not exist"
Nov 03 08:28:09.546994 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:09.546906    5070 scope.go:122] "RemoveContainer" containerID="9a349cac83afa17a6fd7dcd2183a4c8615aecc6cdb9c5ff2a4897155383bcb1f"
Nov 03 08:28:09.549990 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:09.549944    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t" podStartSLOduration=1.54993023 podStartE2EDuration="1.54993023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:28:09.547588604 +0000 UTC m=+2087.662052561" watchObservedRunningTime="2026-11-03 08:28:09.54993023 +0000 UTC m=+2087.664394161"
Nov 03 08:28:09.570034 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:09.570015    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"]
Nov 03 08:28:09.575781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:09.575726    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-n4qb5"]
Nov 03 08:28:09.581435 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:09.581413    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg"]
Nov 03 08:28:09.584308 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:09.584286    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-19bb0a-6d46b697f-f65wg"]
Nov 03 08:28:10.024971 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:10.024940    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2bafc12-e2f3-46ee-8f98-01e46c2c2787" path="/var/lib/kubelet/pods/a2bafc12-e2f3-46ee-8f98-01e46c2c2787/volumes"
Nov 03 08:28:10.025850 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:10.025835    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1a356ec-fba8-45e1-87ba-13950f3e1fcb" path="/var/lib/kubelet/pods/d1a356ec-fba8-45e1-87ba-13950f3e1fcb/volumes"
Nov 03 08:28:10.550696 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:10.550662    5070 generic.go:356] "Generic (PLEG): container finished" podID="961562d5-1288-4dbc-9723-451021080e30" containerID="0999cd513af3becb230dfc5833512911040d67852d69a7e4fba157333c7ab733" exitCode=0
Nov 03 08:28:10.551142 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:10.550747    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t" event={"ID":"961562d5-1288-4dbc-9723-451021080e30","Type":"ContainerDied","Data":"0999cd513af3becb230dfc5833512911040d67852d69a7e4fba157333c7ab733"}
Nov 03 08:28:11.559481 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:11.559446    5070 generic.go:356] "Generic (PLEG): container finished" podID="11074ce5-9baa-4b3b-ae8b-b30409e5efbb" containerID="9a8b7c354325dac7f58d23b3467952b4983a5d2ac30a25c9c7642e085d01de75" exitCode=0
Nov 03 08:28:11.559938 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:11.559518    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst" event={"ID":"11074ce5-9baa-4b3b-ae8b-b30409e5efbb","Type":"ContainerDied","Data":"9a8b7c354325dac7f58d23b3467952b4983a5d2ac30a25c9c7642e085d01de75"}
Nov 03 08:28:11.559938 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:11.559926    5070 scope.go:122] "RemoveContainer" containerID="9a8b7c354325dac7f58d23b3467952b4983a5d2ac30a25c9c7642e085d01de75"
Nov 03 08:28:11.561301 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:11.561268    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:28:11.561380 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:11.561332    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:28:11.561380 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:11.561365    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:28:11.735793 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:11.735388    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t"
Nov 03 08:28:11.787177 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:11.787146    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/961562d5-1288-4dbc-9723-451021080e30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/961562d5-1288-4dbc-9723-451021080e30-maas-api-xks-ca\") pod \"961562d5-1288-4dbc-9723-451021080e30\" (UID: \"961562d5-1288-4dbc-9723-451021080e30\") "
Nov 03 08:28:11.787375 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:11.787194    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/961562d5-1288-4dbc-9723-451021080e30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/961562d5-1288-4dbc-9723-451021080e30-maas-api-openshift-ca\") pod \"961562d5-1288-4dbc-9723-451021080e30\" (UID: \"961562d5-1288-4dbc-9723-451021080e30\") "
Nov 03 08:28:11.787464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:11.787411    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961562d5-1288-4dbc-9723-451021080e30-kube-api-access-w46nr\" (UniqueName: \"kubernetes.io/projected/961562d5-1288-4dbc-9723-451021080e30-kube-api-access-w46nr\") pod \"961562d5-1288-4dbc-9723-451021080e30\" (UID: \"961562d5-1288-4dbc-9723-451021080e30\") "
Nov 03 08:28:11.787937 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:11.787672    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961562d5-1288-4dbc-9723-451021080e30-maas-api-openshift-ca" pod "961562d5-1288-4dbc-9723-451021080e30" (UID: "961562d5-1288-4dbc-9723-451021080e30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:11.787937 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:11.787917    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/961562d5-1288-4dbc-9723-451021080e30-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:11.789976 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:11.789943    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/961562d5-1288-4dbc-9723-451021080e30-maas-api-xks-ca" pod "961562d5-1288-4dbc-9723-451021080e30" (UID: "961562d5-1288-4dbc-9723-451021080e30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:11.790149 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:11.790130    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961562d5-1288-4dbc-9723-451021080e30-kube-api-access-w46nr" pod "961562d5-1288-4dbc-9723-451021080e30" (UID: "961562d5-1288-4dbc-9723-451021080e30"). InnerVolumeSpecName "kube-api-access-w46nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:11.889286 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:11.889249    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/961562d5-1288-4dbc-9723-451021080e30-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:11.889286 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:11.889282    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w46nr\" (UniqueName: \"kubernetes.io/projected/961562d5-1288-4dbc-9723-451021080e30-kube-api-access-w46nr\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:12.201928 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.201900    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:28:12.297730 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.297692    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-kube-api-access-ks9ln\" (UniqueName: \"kubernetes.io/projected/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-kube-api-access-ks9ln\") pod \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") "
Nov 03 08:28:12.297959 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.297790    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-workload-certs\") pod \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") "
Nov 03 08:28:12.297959 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.297900    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-token\" (UniqueName: \"kubernetes.io/projected/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-token\") pod \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") "
Nov 03 08:28:12.297959 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.297947    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-podinfo\") pod \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") "
Nov 03 08:28:12.298151 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.297981    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-envoy\") pod \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") "
Nov 03 08:28:12.298151 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.298075    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-workload-socket\") pod \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") "
Nov 03 08:28:12.298261 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.298200    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-data\") pod \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") "
Nov 03 08:28:12.298321 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.298303    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istiod-ca-cert\") pod \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") "
Nov 03 08:28:12.298396 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.298379    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-credential-socket\") pod \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\" (UID: \"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc\") "
Nov 03 08:28:12.299122 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.299079    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-credential-socket" pod "f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" (UID: "f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:12.299499 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.299472    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-data" pod "f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" (UID: "f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:12.299739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.299691    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-workload-socket" pod "f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" (UID: "f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:12.300328 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.300280    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-workload-certs" pod "f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" (UID: "f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:12.301138 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.301115    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istiod-ca-cert" pod "f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" (UID: "f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:12.303568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.303532    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-podinfo" pod "f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" (UID: "f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:28:12.305554 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.305531    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-envoy" pod "f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" (UID: "f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:12.305858 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.305739    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-kube-api-access-ks9ln" pod "f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" (UID: "f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc"). InnerVolumeSpecName "kube-api-access-ks9ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:12.305858 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.305832    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-token" pod "f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" (UID: "f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:12.381253 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.381224    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"]
Nov 03 08:28:12.400138 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.400109    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:12.400138 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.400139    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks9ln\" (UniqueName: \"kubernetes.io/projected/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-kube-api-access-ks9ln\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:12.400323 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.400153    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:12.400323 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.400165    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:12.400323 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.400177    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:12.400323 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.400189    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:12.400323 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.400201    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:12.400323 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.400214    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:12.400323 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.400227    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:12.577182 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.577080    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst" event={"ID":"11074ce5-9baa-4b3b-ae8b-b30409e5efbb","Type":"ContainerStarted","Data":"d567c7b8e54b3cd9a44781dc93da9561ecc1a1c9ae61df0bad60ee19dd4762ee"}
Nov 03 08:28:12.577639 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.577240    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst" podUID="11074ce5-9baa-4b3b-ae8b-b30409e5efbb" containerName="istio-proxy" containerID="cri-o://d567c7b8e54b3cd9a44781dc93da9561ecc1a1c9ae61df0bad60ee19dd4762ee" gracePeriod=30
Nov 03 08:28:12.578808 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.578788    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t"
Nov 03 08:28:12.578908 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.578786    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t" event={"ID":"961562d5-1288-4dbc-9723-451021080e30","Type":"ContainerDied","Data":"a38b83a441097e99ea392de447a8d9a4023bce5e06de5fe1b6ce84ed53b27717"}
Nov 03 08:28:12.578971 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.578908    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a38b83a441097e99ea392de447a8d9a4023bce5e06de5fe1b6ce84ed53b27717"
Nov 03 08:28:12.582107 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.582073    5070 generic.go:356] "Generic (PLEG): container finished" podID="f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" containerID="9c17e76f83d02982413d20c13e208ed6683938d3c14d74b497e6d9aa053b41ff" exitCode=0
Nov 03 08:28:12.582229 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.582158    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt" event={"ID":"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc","Type":"ContainerDied","Data":"9c17e76f83d02982413d20c13e208ed6683938d3c14d74b497e6d9aa053b41ff"}
Nov 03 08:28:12.582229 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.582178    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"
Nov 03 08:28:12.582356 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.582195    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt" event={"ID":"f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc","Type":"ContainerDied","Data":"213f6107d343f3c6243d27a919654489b0b46e70bde2aac52386cdf099109372"}
Nov 03 08:28:12.582356 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.582255    5070 scope.go:122] "RemoveContainer" containerID="9c17e76f83d02982413d20c13e208ed6683938d3c14d74b497e6d9aa053b41ff"
Nov 03 08:28:12.594475 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.594448    5070 scope.go:122] "RemoveContainer" containerID="dee6741d7474f0befcf7b11602ae9c3b830feec2b33134e736cb84c8c8cac57e"
Nov 03 08:28:12.605628 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.605609    5070 scope.go:122] "RemoveContainer" containerID="9c17e76f83d02982413d20c13e208ed6683938d3c14d74b497e6d9aa053b41ff"
Nov 03 08:28:12.605911 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:28:12.605893    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c17e76f83d02982413d20c13e208ed6683938d3c14d74b497e6d9aa053b41ff\": container with ID starting with 9c17e76f83d02982413d20c13e208ed6683938d3c14d74b497e6d9aa053b41ff not found: ID does not exist" containerID="9c17e76f83d02982413d20c13e208ed6683938d3c14d74b497e6d9aa053b41ff"
Nov 03 08:28:12.605957 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.605918    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c17e76f83d02982413d20c13e208ed6683938d3c14d74b497e6d9aa053b41ff"} err="failed to get container status \"9c17e76f83d02982413d20c13e208ed6683938d3c14d74b497e6d9aa053b41ff\": rpc error: code = NotFound desc = could not find container \"9c17e76f83d02982413d20c13e208ed6683938d3c14d74b497e6d9aa053b41ff\": container with ID starting with 9c17e76f83d02982413d20c13e208ed6683938d3c14d74b497e6d9aa053b41ff not found: ID does not exist"
Nov 03 08:28:12.605957 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.605935    5070 scope.go:122] "RemoveContainer" containerID="dee6741d7474f0befcf7b11602ae9c3b830feec2b33134e736cb84c8c8cac57e"
Nov 03 08:28:12.606175 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:28:12.606156    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dee6741d7474f0befcf7b11602ae9c3b830feec2b33134e736cb84c8c8cac57e\": container with ID starting with dee6741d7474f0befcf7b11602ae9c3b830feec2b33134e736cb84c8c8cac57e not found: ID does not exist" containerID="dee6741d7474f0befcf7b11602ae9c3b830feec2b33134e736cb84c8c8cac57e"
Nov 03 08:28:12.606218 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.606180    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dee6741d7474f0befcf7b11602ae9c3b830feec2b33134e736cb84c8c8cac57e"} err="failed to get container status \"dee6741d7474f0befcf7b11602ae9c3b830feec2b33134e736cb84c8c8cac57e\": rpc error: code = NotFound desc = could not find container \"dee6741d7474f0befcf7b11602ae9c3b830feec2b33134e736cb84c8c8cac57e\": container with ID starting with dee6741d7474f0befcf7b11602ae9c3b830feec2b33134e736cb84c8c8cac57e not found: ID does not exist"
Nov 03 08:28:12.610962 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.610939    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"]
Nov 03 08:28:12.615226 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.615206    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-19bb0a-openshift-default-6cb4c45bf-6nmwt"]
Nov 03 08:28:12.849054 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.848992    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6"]
Nov 03 08:28:12.849490 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.849476    5070 cpu_manager.go:414] "containerMap: removing container" podUID="d1a356ec-fba8-45e1-87ba-13950f3e1fcb" containerName="istio-proxy"
Nov 03 08:28:12.849533 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.849492    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a356ec-fba8-45e1-87ba-13950f3e1fcb" containerName="istio-proxy"
Nov 03 08:28:12.849533 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.849509    5070 cpu_manager.go:414] "containerMap: removing container" podUID="f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" containerName="istio-proxy"
Nov 03 08:28:12.849533 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.849514    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" containerName="istio-proxy"
Nov 03 08:28:12.849533 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.849523    5070 cpu_manager.go:414] "containerMap: removing container" podUID="f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" containerName="istio-proxy"
Nov 03 08:28:12.849533 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.849528    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" containerName="istio-proxy"
Nov 03 08:28:12.849749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.849548    5070 cpu_manager.go:414] "containerMap: removing container" podUID="961562d5-1288-4dbc-9723-451021080e30" containerName="revoke-keys"
Nov 03 08:28:12.849749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.849553    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961562d5-1288-4dbc-9723-451021080e30" containerName="revoke-keys"
Nov 03 08:28:12.849749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.849570    5070 cpu_manager.go:414] "containerMap: removing container" podUID="a2bafc12-e2f3-46ee-8f98-01e46c2c2787" containerName="payload-processing"
Nov 03 08:28:12.849749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.849575    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2bafc12-e2f3-46ee-8f98-01e46c2c2787" containerName="payload-processing"
Nov 03 08:28:12.849749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.849651    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a356ec-fba8-45e1-87ba-13950f3e1fcb" containerName="istio-proxy"
Nov 03 08:28:12.849749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.849659    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" containerName="istio-proxy"
Nov 03 08:28:12.849749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.849666    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" containerName="istio-proxy"
Nov 03 08:28:12.849749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.849676    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="961562d5-1288-4dbc-9723-451021080e30" containerName="revoke-keys"
Nov 03 08:28:12.849749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.849684    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2bafc12-e2f3-46ee-8f98-01e46c2c2787" containerName="payload-processing"
Nov 03 08:28:12.852982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.852967    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6"
Nov 03 08:28:12.857788 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.857747    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:28:12.857957 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.857931    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:28:12.860122 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.860100    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6"]
Nov 03 08:28:12.861455 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.861435    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:28:12.861556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.861488    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:28:12.904571 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.904548    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6\" (UID: \"00cf30b1-61cc-4e99-b3e6-9919d3f94cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6"
Nov 03 08:28:12.904700 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.904602    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw87b\" (UniqueName: \"kubernetes.io/projected/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5-kube-api-access-kw87b\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6\" (UID: \"00cf30b1-61cc-4e99-b3e6-9919d3f94cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6"
Nov 03 08:28:12.904803 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:12.904750    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6\" (UID: \"00cf30b1-61cc-4e99-b3e6-9919d3f94cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6"
Nov 03 08:28:13.005729 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.005701    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6\" (UID: \"00cf30b1-61cc-4e99-b3e6-9919d3f94cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6"
Nov 03 08:28:13.005875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.005773    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw87b\" (UniqueName: \"kubernetes.io/projected/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5-kube-api-access-kw87b\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6\" (UID: \"00cf30b1-61cc-4e99-b3e6-9919d3f94cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6"
Nov 03 08:28:13.005875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.005844    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6\" (UID: \"00cf30b1-61cc-4e99-b3e6-9919d3f94cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6"
Nov 03 08:28:13.006380 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.006362    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6\" (UID: \"00cf30b1-61cc-4e99-b3e6-9919d3f94cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6"
Nov 03 08:28:13.007914 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.007889    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6\" (UID: \"00cf30b1-61cc-4e99-b3e6-9919d3f94cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6"
Nov 03 08:28:13.015047 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.015024    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw87b\" (UniqueName: \"kubernetes.io/projected/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5-kube-api-access-kw87b\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6\" (UID: \"00cf30b1-61cc-4e99-b3e6-9919d3f94cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6"
Nov 03 08:28:13.024362 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.024340    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-042af12e-cgsh4"]
Nov 03 08:28:13.030384 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.030359    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7"]
Nov 03 08:28:13.035301 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.035281    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-042af12e-cgsh4"]
Nov 03 08:28:13.040152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.040128    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c9d667-wqmm7"]
Nov 03 08:28:13.171934 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.171910    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6"
Nov 03 08:28:13.289599 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.289565    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:28:13.411011 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.410930    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-data\") pod \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") "
Nov 03 08:28:13.411011 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.410970    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-workload-socket\") pod \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") "
Nov 03 08:28:13.411246 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.411040    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-token\" (UniqueName: \"kubernetes.io/projected/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-token\") pod \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") "
Nov 03 08:28:13.411246 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.411069    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-credential-socket\") pod \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") "
Nov 03 08:28:13.411246 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.411098    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-envoy\") pod \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") "
Nov 03 08:28:13.411246 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.411125    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istiod-ca-cert\") pod \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") "
Nov 03 08:28:13.411246 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.411167    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-workload-certs\") pod \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") "
Nov 03 08:28:13.411246 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.411197    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-data" pod "79b2069b-a7c8-45eb-9739-9bf3a1fec044" (UID: "79b2069b-a7c8-45eb-9739-9bf3a1fec044"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:13.411564 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.411257    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-podinfo\") pod \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") "
Nov 03 08:28:13.411564 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.411265    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-workload-socket" pod "79b2069b-a7c8-45eb-9739-9bf3a1fec044" (UID: "79b2069b-a7c8-45eb-9739-9bf3a1fec044"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:13.411564 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.411307    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79b2069b-a7c8-45eb-9739-9bf3a1fec044-kube-api-access-qdlgr\" (UniqueName: \"kubernetes.io/projected/79b2069b-a7c8-45eb-9739-9bf3a1fec044-kube-api-access-qdlgr\") pod \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\" (UID: \"79b2069b-a7c8-45eb-9739-9bf3a1fec044\") "
Nov 03 08:28:13.411564 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.411420    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-credential-socket" pod "79b2069b-a7c8-45eb-9739-9bf3a1fec044" (UID: "79b2069b-a7c8-45eb-9739-9bf3a1fec044"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:13.411564 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.411499    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-workload-certs" pod "79b2069b-a7c8-45eb-9739-9bf3a1fec044" (UID: "79b2069b-a7c8-45eb-9739-9bf3a1fec044"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:13.411564 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.411556    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istiod-ca-cert" pod "79b2069b-a7c8-45eb-9739-9bf3a1fec044" (UID: "79b2069b-a7c8-45eb-9739-9bf3a1fec044"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:13.411924 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.411787    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:13.411924 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.411807    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:13.411924 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.411824    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:13.411924 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.411838    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:13.411924 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.411850    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:13.413491 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.413465    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-envoy" pod "79b2069b-a7c8-45eb-9739-9bf3a1fec044" (UID: "79b2069b-a7c8-45eb-9739-9bf3a1fec044"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:13.413623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.413506    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-podinfo" pod "79b2069b-a7c8-45eb-9739-9bf3a1fec044" (UID: "79b2069b-a7c8-45eb-9739-9bf3a1fec044"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:28:13.413838 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.413817    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-token" pod "79b2069b-a7c8-45eb-9739-9bf3a1fec044" (UID: "79b2069b-a7c8-45eb-9739-9bf3a1fec044"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:13.413928 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.413849    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b2069b-a7c8-45eb-9739-9bf3a1fec044-kube-api-access-qdlgr" pod "79b2069b-a7c8-45eb-9739-9bf3a1fec044" (UID: "79b2069b-a7c8-45eb-9739-9bf3a1fec044"). InnerVolumeSpecName "kube-api-access-qdlgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:13.459996 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:28:13.459974    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00cf30b1_61cc_4e99_b3e6_9919d3f94cd5.slice/crio-a78e517b2810454a4859f4d85eb1c9dda65390472028e7030617eb9e41027476 WatchSource:0}: Error finding container a78e517b2810454a4859f4d85eb1c9dda65390472028e7030617eb9e41027476: Status 404 returned error can't find the container with id a78e517b2810454a4859f4d85eb1c9dda65390472028e7030617eb9e41027476
Nov 03 08:28:13.460689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.460665    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6"]
Nov 03 08:28:13.512586 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.512556    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:13.512586 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.512587    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdlgr\" (UniqueName: \"kubernetes.io/projected/79b2069b-a7c8-45eb-9739-9bf3a1fec044-kube-api-access-qdlgr\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:13.512742 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.512598    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:13.512742 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.512607    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79b2069b-a7c8-45eb-9739-9bf3a1fec044-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:13.593871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.593843    5070 generic.go:356] "Generic (PLEG): container finished" podID="79b2069b-a7c8-45eb-9739-9bf3a1fec044" containerID="772ca4ffdb2c297a843bbf7b384c2a89d75821e380e310aeee6b4907ffe91c21" exitCode=0
Nov 03 08:28:13.594275 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.593891    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"
Nov 03 08:28:13.594275 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.593938    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg" event={"ID":"79b2069b-a7c8-45eb-9739-9bf3a1fec044","Type":"ContainerDied","Data":"772ca4ffdb2c297a843bbf7b384c2a89d75821e380e310aeee6b4907ffe91c21"}
Nov 03 08:28:13.594275 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.594006    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg" event={"ID":"79b2069b-a7c8-45eb-9739-9bf3a1fec044","Type":"ContainerDied","Data":"6e90a6ff9ccfa9fcd68bc714061595564feb7ed103a7e6b617b2928a1799457d"}
Nov 03 08:28:13.594275 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.594038    5070 scope.go:122] "RemoveContainer" containerID="772ca4ffdb2c297a843bbf7b384c2a89d75821e380e310aeee6b4907ffe91c21"
Nov 03 08:28:13.595529 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.595501    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6" event={"ID":"00cf30b1-61cc-4e99-b3e6-9919d3f94cd5","Type":"ContainerStarted","Data":"a78e517b2810454a4859f4d85eb1c9dda65390472028e7030617eb9e41027476"}
Nov 03 08:28:13.605718 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.605696    5070 scope.go:122] "RemoveContainer" containerID="2494db5b7c62f02430d1dbc9a883e4ccd091db6f6c76ab1d0159d8b0d5dc8c3f"
Nov 03 08:28:13.617513 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.617488    5070 scope.go:122] "RemoveContainer" containerID="772ca4ffdb2c297a843bbf7b384c2a89d75821e380e310aeee6b4907ffe91c21"
Nov 03 08:28:13.617843 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:28:13.617824    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"772ca4ffdb2c297a843bbf7b384c2a89d75821e380e310aeee6b4907ffe91c21\": container with ID starting with 772ca4ffdb2c297a843bbf7b384c2a89d75821e380e310aeee6b4907ffe91c21 not found: ID does not exist" containerID="772ca4ffdb2c297a843bbf7b384c2a89d75821e380e310aeee6b4907ffe91c21"
Nov 03 08:28:13.617947 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.617849    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"772ca4ffdb2c297a843bbf7b384c2a89d75821e380e310aeee6b4907ffe91c21"} err="failed to get container status \"772ca4ffdb2c297a843bbf7b384c2a89d75821e380e310aeee6b4907ffe91c21\": rpc error: code = NotFound desc = could not find container \"772ca4ffdb2c297a843bbf7b384c2a89d75821e380e310aeee6b4907ffe91c21\": container with ID starting with 772ca4ffdb2c297a843bbf7b384c2a89d75821e380e310aeee6b4907ffe91c21 not found: ID does not exist"
Nov 03 08:28:13.617947 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.617865    5070 scope.go:122] "RemoveContainer" containerID="2494db5b7c62f02430d1dbc9a883e4ccd091db6f6c76ab1d0159d8b0d5dc8c3f"
Nov 03 08:28:13.618126 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:28:13.618108    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2494db5b7c62f02430d1dbc9a883e4ccd091db6f6c76ab1d0159d8b0d5dc8c3f\": container with ID starting with 2494db5b7c62f02430d1dbc9a883e4ccd091db6f6c76ab1d0159d8b0d5dc8c3f not found: ID does not exist" containerID="2494db5b7c62f02430d1dbc9a883e4ccd091db6f6c76ab1d0159d8b0d5dc8c3f"
Nov 03 08:28:13.618183 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.618132    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2494db5b7c62f02430d1dbc9a883e4ccd091db6f6c76ab1d0159d8b0d5dc8c3f"} err="failed to get container status \"2494db5b7c62f02430d1dbc9a883e4ccd091db6f6c76ab1d0159d8b0d5dc8c3f\": rpc error: code = NotFound desc = could not find container \"2494db5b7c62f02430d1dbc9a883e4ccd091db6f6c76ab1d0159d8b0d5dc8c3f\": container with ID starting with 2494db5b7c62f02430d1dbc9a883e4ccd091db6f6c76ab1d0159d8b0d5dc8c3f not found: ID does not exist"
Nov 03 08:28:13.618404 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.618390    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"]
Nov 03 08:28:13.621723 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:13.621699    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0fc79b-openshift-default-7667f77fdf-m2pjg"]
Nov 03 08:28:14.025114 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:14.025057    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="304875a7-473f-4fbf-99ba-8548ac801eb5" path="/var/lib/kubelet/pods/304875a7-473f-4fbf-99ba-8548ac801eb5/volumes"
Nov 03 08:28:14.025520 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:14.025505    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79b2069b-a7c8-45eb-9739-9bf3a1fec044" path="/var/lib/kubelet/pods/79b2069b-a7c8-45eb-9739-9bf3a1fec044/volumes"
Nov 03 08:28:14.025917 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:14.025903    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce2843e0-e7f1-4e37-ba01-11599705c09e" path="/var/lib/kubelet/pods/ce2843e0-e7f1-4e37-ba01-11599705c09e/volumes"
Nov 03 08:28:14.026628 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:14.026613    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc" path="/var/lib/kubelet/pods/f0fa1f2b-15c4-48a8-9eef-d87bbcc6aafc/volumes"
Nov 03 08:28:14.469071 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:14.469040    5070 scope.go:122] "RemoveContainer" containerID="4e95d18141a4fcbb840da5b6169fce01816da57b01ab9effe9d6e517fe26fe13"
Nov 03 08:28:14.479144 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:14.479124    5070 scope.go:122] "RemoveContainer" containerID="4692f1ed69e37cbd8044180ef3ed595a10bbb7b09705fbaff707a06c14230bef"
Nov 03 08:28:14.488505 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:14.488488    5070 scope.go:122] "RemoveContainer" containerID="aa953843af2fe6ebf8b22a847ddcb29b528c15a77f75d0c52963da6a8f9b93ab"
Nov 03 08:28:14.498043 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:14.498021    5070 scope.go:122] "RemoveContainer" containerID="79aca1a1021c1bb1cf4fc500127cb274a1d12f6a454e58edf89f65f133fbe158"
Nov 03 08:28:14.507914 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:14.507898    5070 scope.go:122] "RemoveContainer" containerID="b229960f18c29960c64d79f8cc477319882eed5389c002ebcaf0e0c8ca6d66e0"
Nov 03 08:28:14.606226 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:14.606068    5070 generic.go:356] "Generic (PLEG): container finished" podID="00cf30b1-61cc-4e99-b3e6-9919d3f94cd5" containerID="8c54ba82756debfd77e137822265b74fcb5bd18569130a897749e4d1789481b4" exitCode=0
Nov 03 08:28:14.606226 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:14.606129    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6" event={"ID":"00cf30b1-61cc-4e99-b3e6-9919d3f94cd5","Type":"ContainerDied","Data":"8c54ba82756debfd77e137822265b74fcb5bd18569130a897749e4d1789481b4"}
Nov 03 08:28:15.753680 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:15.753654    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6"
Nov 03 08:28:15.764210 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:15.764176    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:28:15.835399 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:15.835373    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5-maas-api-xks-ca\") pod \"00cf30b1-61cc-4e99-b3e6-9919d3f94cd5\" (UID: \"00cf30b1-61cc-4e99-b3e6-9919d3f94cd5\") "
Nov 03 08:28:15.835540 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:15.835433    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5-kube-api-access-kw87b\" (UniqueName: \"kubernetes.io/projected/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5-kube-api-access-kw87b\") pod \"00cf30b1-61cc-4e99-b3e6-9919d3f94cd5\" (UID: \"00cf30b1-61cc-4e99-b3e6-9919d3f94cd5\") "
Nov 03 08:28:15.835615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:15.835546    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5-maas-api-openshift-ca\") pod \"00cf30b1-61cc-4e99-b3e6-9919d3f94cd5\" (UID: \"00cf30b1-61cc-4e99-b3e6-9919d3f94cd5\") "
Nov 03 08:28:15.835982 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:15.835948    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5-maas-api-openshift-ca" pod "00cf30b1-61cc-4e99-b3e6-9919d3f94cd5" (UID: "00cf30b1-61cc-4e99-b3e6-9919d3f94cd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:15.837481 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:15.837456    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5-maas-api-xks-ca" pod "00cf30b1-61cc-4e99-b3e6-9919d3f94cd5" (UID: "00cf30b1-61cc-4e99-b3e6-9919d3f94cd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:15.837963 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:15.837945    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5-kube-api-access-kw87b" pod "00cf30b1-61cc-4e99-b3e6-9919d3f94cd5" (UID: "00cf30b1-61cc-4e99-b3e6-9919d3f94cd5"). InnerVolumeSpecName "kube-api-access-kw87b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:15.937141 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:15.937079    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:15.937141 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:15.937106    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw87b\" (UniqueName: \"kubernetes.io/projected/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5-kube-api-access-kw87b\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:15.937141 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:15.937117    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:16.622201 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:16.622168    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6"
Nov 03 08:28:16.622352 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:16.622196    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6" event={"ID":"00cf30b1-61cc-4e99-b3e6-9919d3f94cd5","Type":"ContainerDied","Data":"a78e517b2810454a4859f4d85eb1c9dda65390472028e7030617eb9e41027476"}
Nov 03 08:28:16.622352 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:16.622242    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a78e517b2810454a4859f4d85eb1c9dda65390472028e7030617eb9e41027476"
Nov 03 08:28:17.836316 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.836266    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:28:17.960385 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.960360    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-kube-api-access-9264d\" (UniqueName: \"kubernetes.io/projected/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-kube-api-access-9264d\") pod \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") "
Nov 03 08:28:17.960540 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.960409    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istiod-ca-cert\") pod \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") "
Nov 03 08:28:17.960540 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.960476    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-workload-certs\") pod \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") "
Nov 03 08:28:17.960540 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.960494    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-token\" (UniqueName: \"kubernetes.io/projected/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-token\") pod \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") "
Nov 03 08:28:17.960540 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.960533    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-workload-socket\") pod \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") "
Nov 03 08:28:17.960739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.960592    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-envoy\") pod \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") "
Nov 03 08:28:17.960739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.960626    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-podinfo\") pod \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") "
Nov 03 08:28:17.960739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.960654    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-credential-socket\") pod \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") "
Nov 03 08:28:17.960739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.960727    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-data\") pod \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\" (UID: \"11074ce5-9baa-4b3b-ae8b-b30409e5efbb\") "
Nov 03 08:28:17.961019 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.960913    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istiod-ca-cert" pod "11074ce5-9baa-4b3b-ae8b-b30409e5efbb" (UID: "11074ce5-9baa-4b3b-ae8b-b30409e5efbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:17.961152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.961125    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-workload-certs" pod "11074ce5-9baa-4b3b-ae8b-b30409e5efbb" (UID: "11074ce5-9baa-4b3b-ae8b-b30409e5efbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:17.961246 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.960966    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-workload-socket" pod "11074ce5-9baa-4b3b-ae8b-b30409e5efbb" (UID: "11074ce5-9baa-4b3b-ae8b-b30409e5efbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:17.961246 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.961191    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-data" pod "11074ce5-9baa-4b3b-ae8b-b30409e5efbb" (UID: "11074ce5-9baa-4b3b-ae8b-b30409e5efbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:17.961246 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.961226    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-credential-socket" pod "11074ce5-9baa-4b3b-ae8b-b30409e5efbb" (UID: "11074ce5-9baa-4b3b-ae8b-b30409e5efbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:17.961370 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.961315    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:17.961370 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.961328    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:17.961370 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.961338    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:17.961370 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.961347    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:17.962916 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.962884    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-envoy" pod "11074ce5-9baa-4b3b-ae8b-b30409e5efbb" (UID: "11074ce5-9baa-4b3b-ae8b-b30409e5efbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:17.963242 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.963215    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-token" pod "11074ce5-9baa-4b3b-ae8b-b30409e5efbb" (UID: "11074ce5-9baa-4b3b-ae8b-b30409e5efbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:17.963242 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.963224    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-kube-api-access-9264d" pod "11074ce5-9baa-4b3b-ae8b-b30409e5efbb" (UID: "11074ce5-9baa-4b3b-ae8b-b30409e5efbb"). InnerVolumeSpecName "kube-api-access-9264d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:17.963242 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:17.963221    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-podinfo" pod "11074ce5-9baa-4b3b-ae8b-b30409e5efbb" (UID: "11074ce5-9baa-4b3b-ae8b-b30409e5efbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:28:18.061978 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:18.061951    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:18.062070 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:18.061980    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:18.062070 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:18.061995    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:18.062070 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:18.062010    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:18.062070 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:18.062024    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9264d\" (UniqueName: \"kubernetes.io/projected/11074ce5-9baa-4b3b-ae8b-b30409e5efbb-kube-api-access-9264d\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:18.640293 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:18.640252    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t"]
Nov 03 08:28:18.641052 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:18.641024    5070 generic.go:356] "Generic (PLEG): container finished" podID="11074ce5-9baa-4b3b-ae8b-b30409e5efbb" containerID="d567c7b8e54b3cd9a44781dc93da9561ecc1a1c9ae61df0bad60ee19dd4762ee" exitCode=0
Nov 03 08:28:18.641162 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:18.641078    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"
Nov 03 08:28:18.641162 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:18.641103    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst" event={"ID":"11074ce5-9baa-4b3b-ae8b-b30409e5efbb","Type":"ContainerDied","Data":"d567c7b8e54b3cd9a44781dc93da9561ecc1a1c9ae61df0bad60ee19dd4762ee"}
Nov 03 08:28:18.641242 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:18.641166    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst" event={"ID":"11074ce5-9baa-4b3b-ae8b-b30409e5efbb","Type":"ContainerDied","Data":"32f67b455818f6d12cca9a55a2d2ed278f21d26996f8f34ffd7f6d5dfe69f481"}
Nov 03 08:28:18.641242 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:18.641197    5070 scope.go:122] "RemoveContainer" containerID="d567c7b8e54b3cd9a44781dc93da9561ecc1a1c9ae61df0bad60ee19dd4762ee"
Nov 03 08:28:18.651456 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:18.651435    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk"]
Nov 03 08:28:18.653395 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:18.653374    5070 scope.go:122] "RemoveContainer" containerID="9a8b7c354325dac7f58d23b3467952b4983a5d2ac30a25c9c7642e085d01de75"
Nov 03 08:28:18.667372 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:18.667281    5070 scope.go:122] "RemoveContainer" containerID="d567c7b8e54b3cd9a44781dc93da9561ecc1a1c9ae61df0bad60ee19dd4762ee"
Nov 03 08:28:18.667689 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:28:18.667602    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d567c7b8e54b3cd9a44781dc93da9561ecc1a1c9ae61df0bad60ee19dd4762ee\": container with ID starting with d567c7b8e54b3cd9a44781dc93da9561ecc1a1c9ae61df0bad60ee19dd4762ee not found: ID does not exist" containerID="d567c7b8e54b3cd9a44781dc93da9561ecc1a1c9ae61df0bad60ee19dd4762ee"
Nov 03 08:28:18.667689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:18.667634    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d567c7b8e54b3cd9a44781dc93da9561ecc1a1c9ae61df0bad60ee19dd4762ee"} err="failed to get container status \"d567c7b8e54b3cd9a44781dc93da9561ecc1a1c9ae61df0bad60ee19dd4762ee\": rpc error: code = NotFound desc = could not find container \"d567c7b8e54b3cd9a44781dc93da9561ecc1a1c9ae61df0bad60ee19dd4762ee\": container with ID starting with d567c7b8e54b3cd9a44781dc93da9561ecc1a1c9ae61df0bad60ee19dd4762ee not found: ID does not exist"
Nov 03 08:28:18.667689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:18.667659    5070 scope.go:122] "RemoveContainer" containerID="9a8b7c354325dac7f58d23b3467952b4983a5d2ac30a25c9c7642e085d01de75"
Nov 03 08:28:18.669133 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:28:18.668315    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a8b7c354325dac7f58d23b3467952b4983a5d2ac30a25c9c7642e085d01de75\": container with ID starting with 9a8b7c354325dac7f58d23b3467952b4983a5d2ac30a25c9c7642e085d01de75 not found: ID does not exist" containerID="9a8b7c354325dac7f58d23b3467952b4983a5d2ac30a25c9c7642e085d01de75"
Nov 03 08:28:18.669133 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:18.668344    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a8b7c354325dac7f58d23b3467952b4983a5d2ac30a25c9c7642e085d01de75"} err="failed to get container status \"9a8b7c354325dac7f58d23b3467952b4983a5d2ac30a25c9c7642e085d01de75\": rpc error: code = NotFound desc = could not find container \"9a8b7c354325dac7f58d23b3467952b4983a5d2ac30a25c9c7642e085d01de75\": container with ID starting with 9a8b7c354325dac7f58d23b3467952b4983a5d2ac30a25c9c7642e085d01de75 not found: ID does not exist"
Nov 03 08:28:18.673400 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:18.672909    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"]
Nov 03 08:28:18.689614 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:18.689592    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-178e68-openshift-default-6664f8fd65-gqwst"]
Nov 03 08:28:19.337280 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:19.337246    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t"]
Nov 03 08:28:19.340497 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:19.340477    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d168e-nf98t"]
Nov 03 08:28:20.026505 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:20.026467    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11074ce5-9baa-4b3b-ae8b-b30409e5efbb" path="/var/lib/kubelet/pods/11074ce5-9baa-4b3b-ae8b-b30409e5efbb/volumes"
Nov 03 08:28:20.026939 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:20.026924    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="961562d5-1288-4dbc-9723-451021080e30" path="/var/lib/kubelet/pods/961562d5-1288-4dbc-9723-451021080e30/volumes"
Nov 03 08:28:22.050427 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:28:22.050393    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:28:22.607680 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:22.607654    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/3.log"
Nov 03 08:28:22.608961 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:22.608938    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/3.log"
Nov 03 08:28:22.616247 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:22.616225    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-n28db_5738916f-8202-40ed-823e-90f7c9cc4ec9/csi-driver/0.log"
Nov 03 08:28:22.617345 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:22.617327    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-n28db_5738916f-8202-40ed-823e-90f7c9cc4ec9/csi-driver/0.log"
Nov 03 08:28:23.641591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:23.641549    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t" podUID="216f610f-a3c5-4967-949d-fbabca17ed1a" containerName="payload-processing" containerID="cri-o://b6361dc0082e8ba4e47b63f2a172f96825a618ffec32652a95f141a96a591727" gracePeriod=25
Nov 03 08:28:23.652156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:23.652111    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk" podUID="4fcd6530-2913-45a2-a36b-697f56d42909" containerName="payload-pre-processing" containerID="cri-o://7f5cb43858694e81fd3723a628ef723d32fff0f13de95c05b81def5ae0a0203e" gracePeriod=25
Nov 03 08:28:25.751096 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:25.751059    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6"]
Nov 03 08:28:25.755838 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:25.755812    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ed5d1a-lfsq6"]
Nov 03 08:28:26.027224 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:26.027146    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00cf30b1-61cc-4e99-b3e6-9919d3f94cd5" path="/var/lib/kubelet/pods/00cf30b1-61cc-4e99-b3e6-9919d3f94cd5/volumes"
Nov 03 08:28:26.027479 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:26.027464    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44"]
Nov 03 08:28:26.027521 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:26.027496    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2426787c-cwn44"]
Nov 03 08:28:27.631669 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.631639    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf"]
Nov 03 08:28:27.632364 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.632345    5070 cpu_manager.go:414] "containerMap: removing container" podUID="79b2069b-a7c8-45eb-9739-9bf3a1fec044" containerName="istio-proxy"
Nov 03 08:28:27.632416 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.632367    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b2069b-a7c8-45eb-9739-9bf3a1fec044" containerName="istio-proxy"
Nov 03 08:28:27.632453 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.632414    5070 cpu_manager.go:414] "containerMap: removing container" podUID="11074ce5-9baa-4b3b-ae8b-b30409e5efbb" containerName="istio-proxy"
Nov 03 08:28:27.632453 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.632422    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11074ce5-9baa-4b3b-ae8b-b30409e5efbb" containerName="istio-proxy"
Nov 03 08:28:27.632453 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.632444    5070 cpu_manager.go:414] "containerMap: removing container" podUID="11074ce5-9baa-4b3b-ae8b-b30409e5efbb" containerName="istio-proxy"
Nov 03 08:28:27.632453 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.632452    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11074ce5-9baa-4b3b-ae8b-b30409e5efbb" containerName="istio-proxy"
Nov 03 08:28:27.632590 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.632474    5070 cpu_manager.go:414] "containerMap: removing container" podUID="00cf30b1-61cc-4e99-b3e6-9919d3f94cd5" containerName="revoke-keys"
Nov 03 08:28:27.632590 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.632481    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00cf30b1-61cc-4e99-b3e6-9919d3f94cd5" containerName="revoke-keys"
Nov 03 08:28:27.632590 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.632585    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="11074ce5-9baa-4b3b-ae8b-b30409e5efbb" containerName="istio-proxy"
Nov 03 08:28:27.632687 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.632603    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="00cf30b1-61cc-4e99-b3e6-9919d3f94cd5" containerName="revoke-keys"
Nov 03 08:28:27.632687 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.632614    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b2069b-a7c8-45eb-9739-9bf3a1fec044" containerName="istio-proxy"
Nov 03 08:28:27.632687 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.632624    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="79b2069b-a7c8-45eb-9739-9bf3a1fec044" containerName="istio-proxy"
Nov 03 08:28:27.637585 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.637565    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf"
Nov 03 08:28:27.641724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.641699    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:28:27.641873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.641837    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:28:27.645287 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.645263    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf"]
Nov 03 08:28:27.648333 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.648309    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:28:27.648467 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.648394    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:28:27.765720 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.765692    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m5bw\" (UniqueName: \"kubernetes.io/projected/e01352d1-5185-4b50-9868-d2b0b5e38890-kube-api-access-9m5bw\") pod \"maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf\" (UID: \"e01352d1-5185-4b50-9868-d2b0b5e38890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf"
Nov 03 08:28:27.765920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.765802    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e01352d1-5185-4b50-9868-d2b0b5e38890-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf\" (UID: \"e01352d1-5185-4b50-9868-d2b0b5e38890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf"
Nov 03 08:28:27.765920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.765874    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e01352d1-5185-4b50-9868-d2b0b5e38890-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf\" (UID: \"e01352d1-5185-4b50-9868-d2b0b5e38890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf"
Nov 03 08:28:27.867268 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.867237    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e01352d1-5185-4b50-9868-d2b0b5e38890-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf\" (UID: \"e01352d1-5185-4b50-9868-d2b0b5e38890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf"
Nov 03 08:28:27.867464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.867277    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e01352d1-5185-4b50-9868-d2b0b5e38890-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf\" (UID: \"e01352d1-5185-4b50-9868-d2b0b5e38890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf"
Nov 03 08:28:27.867464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.867404    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m5bw\" (UniqueName: \"kubernetes.io/projected/e01352d1-5185-4b50-9868-d2b0b5e38890-kube-api-access-9m5bw\") pod \"maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf\" (UID: \"e01352d1-5185-4b50-9868-d2b0b5e38890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf"
Nov 03 08:28:27.868047 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.868024    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e01352d1-5185-4b50-9868-d2b0b5e38890-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf\" (UID: \"e01352d1-5185-4b50-9868-d2b0b5e38890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf"
Nov 03 08:28:27.869571 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.869552    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e01352d1-5185-4b50-9868-d2b0b5e38890-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf\" (UID: \"e01352d1-5185-4b50-9868-d2b0b5e38890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf"
Nov 03 08:28:27.874297 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.874275    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m5bw\" (UniqueName: \"kubernetes.io/projected/e01352d1-5185-4b50-9868-d2b0b5e38890-kube-api-access-9m5bw\") pod \"maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf\" (UID: \"e01352d1-5185-4b50-9868-d2b0b5e38890\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf"
Nov 03 08:28:27.956662 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:27.956606    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf"
Nov 03 08:28:28.030405 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:28.030373    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151" path="/var/lib/kubelet/pods/1f1bfff4-be24-4fdf-a2a8-57cc0f2c9151/volumes"
Nov 03 08:28:28.032030 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:28.032006    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh"]
Nov 03 08:28:28.038987 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:28.038960    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d710f944-7jvsh"]
Nov 03 08:28:28.255829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:28.255806    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf"]
Nov 03 08:28:28.257040 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:28:28.257014    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01352d1_5185_4b50_9868_d2b0b5e38890.slice/crio-9a915957986384cf6513c7d467c71005a1665adbe981b1716093d252f3d5bd29 WatchSource:0}: Error finding container 9a915957986384cf6513c7d467c71005a1665adbe981b1716093d252f3d5bd29: Status 404 returned error can't find the container with id 9a915957986384cf6513c7d467c71005a1665adbe981b1716093d252f3d5bd29
Nov 03 08:28:28.731028 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:28.730995    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf" event={"ID":"e01352d1-5185-4b50-9868-d2b0b5e38890","Type":"ContainerStarted","Data":"9090debcd715791190c07d45effaa2b3fb6c4e966f1eb2ee32caf30b67a4225a"}
Nov 03 08:28:28.731386 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:28.731045    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf" event={"ID":"e01352d1-5185-4b50-9868-d2b0b5e38890","Type":"ContainerStarted","Data":"9a915957986384cf6513c7d467c71005a1665adbe981b1716093d252f3d5bd29"}
Nov 03 08:28:28.747107 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:28.747058    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf" podStartSLOduration=1.747042182 podStartE2EDuration="1.747042182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:28:28.744210665 +0000 UTC m=+2106.858674596" watchObservedRunningTime="2026-11-03 08:28:28.747042182 +0000 UTC m=+2106.861506117"
Nov 03 08:28:28.808610 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:28.808581    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:28:29.740678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:29.740638    5070 generic.go:356] "Generic (PLEG): container finished" podID="e01352d1-5185-4b50-9868-d2b0b5e38890" containerID="9090debcd715791190c07d45effaa2b3fb6c4e966f1eb2ee32caf30b67a4225a" exitCode=0
Nov 03 08:28:29.741094 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:29.740721    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf" event={"ID":"e01352d1-5185-4b50-9868-d2b0b5e38890","Type":"ContainerDied","Data":"9090debcd715791190c07d45effaa2b3fb6c4e966f1eb2ee32caf30b67a4225a"}
Nov 03 08:28:30.027432 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:30.027354    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edddbc48-8787-4a3e-bdce-3f7740681280" path="/var/lib/kubelet/pods/edddbc48-8787-4a3e-bdce-3f7740681280/volumes"
Nov 03 08:28:30.713908 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:30.713863    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t" podUID="216f610f-a3c5-4967-949d-fbabca17ed1a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:28:30.897489 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:30.897464    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf"
Nov 03 08:28:31.003796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:31.003708    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e01352d1-5185-4b50-9868-d2b0b5e38890-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e01352d1-5185-4b50-9868-d2b0b5e38890-maas-api-xks-ca\") pod \"e01352d1-5185-4b50-9868-d2b0b5e38890\" (UID: \"e01352d1-5185-4b50-9868-d2b0b5e38890\") "
Nov 03 08:28:31.003953 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:31.003804    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e01352d1-5185-4b50-9868-d2b0b5e38890-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e01352d1-5185-4b50-9868-d2b0b5e38890-maas-api-openshift-ca\") pod \"e01352d1-5185-4b50-9868-d2b0b5e38890\" (UID: \"e01352d1-5185-4b50-9868-d2b0b5e38890\") "
Nov 03 08:28:31.003953 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:31.003915    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e01352d1-5185-4b50-9868-d2b0b5e38890-kube-api-access-9m5bw\" (UniqueName: \"kubernetes.io/projected/e01352d1-5185-4b50-9868-d2b0b5e38890-kube-api-access-9m5bw\") pod \"e01352d1-5185-4b50-9868-d2b0b5e38890\" (UID: \"e01352d1-5185-4b50-9868-d2b0b5e38890\") "
Nov 03 08:28:31.004167 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:31.004144    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e01352d1-5185-4b50-9868-d2b0b5e38890-maas-api-openshift-ca" pod "e01352d1-5185-4b50-9868-d2b0b5e38890" (UID: "e01352d1-5185-4b50-9868-d2b0b5e38890"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:31.006026 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:31.005999    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e01352d1-5185-4b50-9868-d2b0b5e38890-maas-api-xks-ca" pod "e01352d1-5185-4b50-9868-d2b0b5e38890" (UID: "e01352d1-5185-4b50-9868-d2b0b5e38890"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:31.006026 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:31.006000    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01352d1-5185-4b50-9868-d2b0b5e38890-kube-api-access-9m5bw" pod "e01352d1-5185-4b50-9868-d2b0b5e38890" (UID: "e01352d1-5185-4b50-9868-d2b0b5e38890"). InnerVolumeSpecName "kube-api-access-9m5bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:31.104967 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:31.104933    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m5bw\" (UniqueName: \"kubernetes.io/projected/e01352d1-5185-4b50-9868-d2b0b5e38890-kube-api-access-9m5bw\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:31.104967 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:31.104960    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e01352d1-5185-4b50-9868-d2b0b5e38890-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:31.104967 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:31.104970    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e01352d1-5185-4b50-9868-d2b0b5e38890-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:31.757031 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:31.757000    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf"
Nov 03 08:28:31.757211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:31.757052    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf" event={"ID":"e01352d1-5185-4b50-9868-d2b0b5e38890","Type":"ContainerDied","Data":"9a915957986384cf6513c7d467c71005a1665adbe981b1716093d252f3d5bd29"}
Nov 03 08:28:31.757211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:31.757101    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a915957986384cf6513c7d467c71005a1665adbe981b1716093d252f3d5bd29"
Nov 03 08:28:32.467109 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:32.467078    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"]
Nov 03 08:28:32.479466 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:32.479434    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"]
Nov 03 08:28:32.737608 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:32.737517    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk" podUID="4fcd6530-2913-45a2-a36b-697f56d42909" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:28:33.451608 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.451577    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9"]
Nov 03 08:28:33.452114 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.452098    5070 cpu_manager.go:414] "containerMap: removing container" podUID="e01352d1-5185-4b50-9868-d2b0b5e38890" containerName="revoke-keys"
Nov 03 08:28:33.452164 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.452115    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e01352d1-5185-4b50-9868-d2b0b5e38890" containerName="revoke-keys"
Nov 03 08:28:33.452164 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.452140    5070 cpu_manager.go:414] "containerMap: removing container" podUID="79b2069b-a7c8-45eb-9739-9bf3a1fec044" containerName="istio-proxy"
Nov 03 08:28:33.452164 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.452145    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79b2069b-a7c8-45eb-9739-9bf3a1fec044" containerName="istio-proxy"
Nov 03 08:28:33.452307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.452249    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="11074ce5-9baa-4b3b-ae8b-b30409e5efbb" containerName="istio-proxy"
Nov 03 08:28:33.452307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.452258    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="e01352d1-5185-4b50-9868-d2b0b5e38890" containerName="revoke-keys"
Nov 03 08:28:33.455419 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.455401    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9"
Nov 03 08:28:33.459293 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.459266    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:28:33.459445 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.459419    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:28:33.459505 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.459472    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:28:33.459595 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.459567    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:28:33.472726 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.465653    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9"]
Nov 03 08:28:33.630030 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.629999    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c92f1c-4491-4136-9800-e0475139651e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9\" (UID: \"51c92f1c-4491-4136-9800-e0475139651e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9"
Nov 03 08:28:33.630202 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.630106    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c92f1c-4491-4136-9800-e0475139651e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9\" (UID: \"51c92f1c-4491-4136-9800-e0475139651e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9"
Nov 03 08:28:33.630202 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.630168    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6p2t\" (UniqueName: \"kubernetes.io/projected/51c92f1c-4491-4136-9800-e0475139651e-kube-api-access-z6p2t\") pod \"maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9\" (UID: \"51c92f1c-4491-4136-9800-e0475139651e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9"
Nov 03 08:28:33.731084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.731015    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c92f1c-4491-4136-9800-e0475139651e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9\" (UID: \"51c92f1c-4491-4136-9800-e0475139651e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9"
Nov 03 08:28:33.731084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.731054    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6p2t\" (UniqueName: \"kubernetes.io/projected/51c92f1c-4491-4136-9800-e0475139651e-kube-api-access-z6p2t\") pod \"maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9\" (UID: \"51c92f1c-4491-4136-9800-e0475139651e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9"
Nov 03 08:28:33.731304 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.731179    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c92f1c-4491-4136-9800-e0475139651e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9\" (UID: \"51c92f1c-4491-4136-9800-e0475139651e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9"
Nov 03 08:28:33.731798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.731776    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c92f1c-4491-4136-9800-e0475139651e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9\" (UID: \"51c92f1c-4491-4136-9800-e0475139651e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9"
Nov 03 08:28:33.733529 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.733505    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c92f1c-4491-4136-9800-e0475139651e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9\" (UID: \"51c92f1c-4491-4136-9800-e0475139651e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9"
Nov 03 08:28:33.738090 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.738065    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6p2t\" (UniqueName: \"kubernetes.io/projected/51c92f1c-4491-4136-9800-e0475139651e-kube-api-access-z6p2t\") pod \"maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9\" (UID: \"51c92f1c-4491-4136-9800-e0475139651e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9"
Nov 03 08:28:33.774804 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:33.774780    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9"
Nov 03 08:28:34.055151 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:34.055118    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9"]
Nov 03 08:28:34.055882 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:28:34.055856    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c92f1c_4491_4136_9800_e0475139651e.slice/crio-8b8658b64196c47591f45f5a054f4ec646b9ccc07e3f9e93b5f5e3291e9e5954 WatchSource:0}: Error finding container 8b8658b64196c47591f45f5a054f4ec646b9ccc07e3f9e93b5f5e3291e9e5954: Status 404 returned error can't find the container with id 8b8658b64196c47591f45f5a054f4ec646b9ccc07e3f9e93b5f5e3291e9e5954
Nov 03 08:28:34.779546 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:34.779454    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9" event={"ID":"51c92f1c-4491-4136-9800-e0475139651e","Type":"ContainerStarted","Data":"12acdd43fcff37bcc1116f28bc0eb56ec458570142a6c8ad8b837b9def88387f"}
Nov 03 08:28:34.779546 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:34.779509    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9" event={"ID":"51c92f1c-4491-4136-9800-e0475139651e","Type":"ContainerStarted","Data":"8b8658b64196c47591f45f5a054f4ec646b9ccc07e3f9e93b5f5e3291e9e5954"}
Nov 03 08:28:34.811863 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:34.811818    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9" podStartSLOduration=1.8118047339999999 podStartE2EDuration="1.811804734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:28:34.809524945 +0000 UTC m=+2112.923988875" watchObservedRunningTime="2026-11-03 08:28:34.811804734 +0000 UTC m=+2112.926268664"
Nov 03 08:28:35.788125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:35.788087    5070 generic.go:356] "Generic (PLEG): container finished" podID="51c92f1c-4491-4136-9800-e0475139651e" containerID="12acdd43fcff37bcc1116f28bc0eb56ec458570142a6c8ad8b837b9def88387f" exitCode=0
Nov 03 08:28:35.788511 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:35.788165    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9" event={"ID":"51c92f1c-4491-4136-9800-e0475139651e","Type":"ContainerDied","Data":"12acdd43fcff37bcc1116f28bc0eb56ec458570142a6c8ad8b837b9def88387f"}
Nov 03 08:28:36.949876 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:36.949851    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9"
Nov 03 08:28:37.065072 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:37.065009    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51c92f1c-4491-4136-9800-e0475139651e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c92f1c-4491-4136-9800-e0475139651e-maas-api-xks-ca\") pod \"51c92f1c-4491-4136-9800-e0475139651e\" (UID: \"51c92f1c-4491-4136-9800-e0475139651e\") "
Nov 03 08:28:37.065229 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:37.065098    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51c92f1c-4491-4136-9800-e0475139651e-kube-api-access-z6p2t\" (UniqueName: \"kubernetes.io/projected/51c92f1c-4491-4136-9800-e0475139651e-kube-api-access-z6p2t\") pod \"51c92f1c-4491-4136-9800-e0475139651e\" (UID: \"51c92f1c-4491-4136-9800-e0475139651e\") "
Nov 03 08:28:37.065229 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:37.065121    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51c92f1c-4491-4136-9800-e0475139651e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c92f1c-4491-4136-9800-e0475139651e-maas-api-openshift-ca\") pod \"51c92f1c-4491-4136-9800-e0475139651e\" (UID: \"51c92f1c-4491-4136-9800-e0475139651e\") "
Nov 03 08:28:37.065553 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:37.065528    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51c92f1c-4491-4136-9800-e0475139651e-maas-api-openshift-ca" pod "51c92f1c-4491-4136-9800-e0475139651e" (UID: "51c92f1c-4491-4136-9800-e0475139651e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:37.067193 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:37.067170    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51c92f1c-4491-4136-9800-e0475139651e-maas-api-xks-ca" pod "51c92f1c-4491-4136-9800-e0475139651e" (UID: "51c92f1c-4491-4136-9800-e0475139651e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:37.067265 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:37.067237    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c92f1c-4491-4136-9800-e0475139651e-kube-api-access-z6p2t" pod "51c92f1c-4491-4136-9800-e0475139651e" (UID: "51c92f1c-4491-4136-9800-e0475139651e"). InnerVolumeSpecName "kube-api-access-z6p2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:37.166643 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:37.166614    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51c92f1c-4491-4136-9800-e0475139651e-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:37.166643 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:37.166639    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51c92f1c-4491-4136-9800-e0475139651e-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:37.166811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:37.166653    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6p2t\" (UniqueName: \"kubernetes.io/projected/51c92f1c-4491-4136-9800-e0475139651e-kube-api-access-z6p2t\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:37.765713 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:37.765676    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9"]
Nov 03 08:28:37.811528 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:37.811496    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9" event={"ID":"51c92f1c-4491-4136-9800-e0475139651e","Type":"ContainerDied","Data":"8b8658b64196c47591f45f5a054f4ec646b9ccc07e3f9e93b5f5e3291e9e5954"}
Nov 03 08:28:37.811730 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:37.811546    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b8658b64196c47591f45f5a054f4ec646b9ccc07e3f9e93b5f5e3291e9e5954"
Nov 03 08:28:37.811730 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:37.811561    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9"
Nov 03 08:28:38.301021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:38.300981    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"]
Nov 03 08:28:39.568952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:39.568918    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr"]
Nov 03 08:28:39.569427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:39.569413    5070 cpu_manager.go:414] "containerMap: removing container" podUID="51c92f1c-4491-4136-9800-e0475139651e" containerName="revoke-keys"
Nov 03 08:28:39.569477 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:39.569427    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51c92f1c-4491-4136-9800-e0475139651e" containerName="revoke-keys"
Nov 03 08:28:39.569522 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:39.569513    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="51c92f1c-4491-4136-9800-e0475139651e" containerName="revoke-keys"
Nov 03 08:28:39.573133 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:39.573117    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr"
Nov 03 08:28:39.582854 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:39.582830    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:28:39.583007 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:39.582871    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:28:39.583007 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:39.582873    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:28:39.583007 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:39.582943    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:28:39.583495 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:39.583473    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr"]
Nov 03 08:28:39.693112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:39.693083    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9hwt\" (UniqueName: \"kubernetes.io/projected/aff7a503-9cf4-4538-8bd5-da7ad324a6af-kube-api-access-f9hwt\") pod \"maas-api-revoke-sub-6249b7ca-zlvsr\" (UID: \"aff7a503-9cf4-4538-8bd5-da7ad324a6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr"
Nov 03 08:28:39.693266 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:39.693191    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff7a503-9cf4-4538-8bd5-da7ad324a6af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6249b7ca-zlvsr\" (UID: \"aff7a503-9cf4-4538-8bd5-da7ad324a6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr"
Nov 03 08:28:39.693266 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:39.693215    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff7a503-9cf4-4538-8bd5-da7ad324a6af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6249b7ca-zlvsr\" (UID: \"aff7a503-9cf4-4538-8bd5-da7ad324a6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr"
Nov 03 08:28:39.794036 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:39.794004    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9hwt\" (UniqueName: \"kubernetes.io/projected/aff7a503-9cf4-4538-8bd5-da7ad324a6af-kube-api-access-f9hwt\") pod \"maas-api-revoke-sub-6249b7ca-zlvsr\" (UID: \"aff7a503-9cf4-4538-8bd5-da7ad324a6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr"
Nov 03 08:28:39.794184 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:39.794127    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff7a503-9cf4-4538-8bd5-da7ad324a6af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6249b7ca-zlvsr\" (UID: \"aff7a503-9cf4-4538-8bd5-da7ad324a6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr"
Nov 03 08:28:39.794184 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:39.794163    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff7a503-9cf4-4538-8bd5-da7ad324a6af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6249b7ca-zlvsr\" (UID: \"aff7a503-9cf4-4538-8bd5-da7ad324a6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr"
Nov 03 08:28:39.794868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:39.794848    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff7a503-9cf4-4538-8bd5-da7ad324a6af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6249b7ca-zlvsr\" (UID: \"aff7a503-9cf4-4538-8bd5-da7ad324a6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr"
Nov 03 08:28:39.796499 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:39.796478    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff7a503-9cf4-4538-8bd5-da7ad324a6af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6249b7ca-zlvsr\" (UID: \"aff7a503-9cf4-4538-8bd5-da7ad324a6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr"
Nov 03 08:28:39.803010 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:39.802986    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9hwt\" (UniqueName: \"kubernetes.io/projected/aff7a503-9cf4-4538-8bd5-da7ad324a6af-kube-api-access-f9hwt\") pod \"maas-api-revoke-sub-6249b7ca-zlvsr\" (UID: \"aff7a503-9cf4-4538-8bd5-da7ad324a6af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr"
Nov 03 08:28:39.888784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:39.888740    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr"
Nov 03 08:28:40.165965 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:40.165934    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr"]
Nov 03 08:28:40.166950 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:28:40.166925    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff7a503_9cf4_4538_8bd5_da7ad324a6af.slice/crio-05e2bb151477c956c0fe9e747ff0ea2b4c1d9ddef51b9f4e8956ea1856bae532 WatchSource:0}: Error finding container 05e2bb151477c956c0fe9e747ff0ea2b4c1d9ddef51b9f4e8956ea1856bae532: Status 404 returned error can't find the container with id 05e2bb151477c956c0fe9e747ff0ea2b4c1d9ddef51b9f4e8956ea1856bae532
Nov 03 08:28:40.713346 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:40.713304    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t" podUID="216f610f-a3c5-4967-949d-fbabca17ed1a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:28:40.834924 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:40.834843    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr" event={"ID":"aff7a503-9cf4-4538-8bd5-da7ad324a6af","Type":"ContainerStarted","Data":"297f7d3cb65aac8da005abcf5293af6d9961600208519db880c01deafeb0c5f7"}
Nov 03 08:28:40.835101 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:40.834930    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr" event={"ID":"aff7a503-9cf4-4538-8bd5-da7ad324a6af","Type":"ContainerStarted","Data":"05e2bb151477c956c0fe9e747ff0ea2b4c1d9ddef51b9f4e8956ea1856bae532"}
Nov 03 08:28:40.853341 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:40.853287    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr" podStartSLOduration=1.853273974 podStartE2EDuration="1.853273974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:28:40.850154543 +0000 UTC m=+2118.964618473" watchObservedRunningTime="2026-11-03 08:28:40.853273974 +0000 UTC m=+2118.967737904"
Nov 03 08:28:41.844287 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:41.844248    5070 generic.go:356] "Generic (PLEG): container finished" podID="aff7a503-9cf4-4538-8bd5-da7ad324a6af" containerID="297f7d3cb65aac8da005abcf5293af6d9961600208519db880c01deafeb0c5f7" exitCode=0
Nov 03 08:28:41.844690 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:41.844309    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr" event={"ID":"aff7a503-9cf4-4538-8bd5-da7ad324a6af","Type":"ContainerDied","Data":"297f7d3cb65aac8da005abcf5293af6d9961600208519db880c01deafeb0c5f7"}
Nov 03 08:28:42.738283 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:42.738239    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk" podUID="4fcd6530-2913-45a2-a36b-697f56d42909" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:28:42.766241 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:42.766196    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9" podUID="917243d5-933a-4f61-9e76-72ec95f6a0dd" containerName="payload-processing" containerID="cri-o://311bea97dce6e955643a8cacfee6d4af6170e9aa483d13c65db7926ed4bd1e30" gracePeriod=25
Nov 03 08:28:43.015444 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.015417    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr"
Nov 03 08:28:43.026101 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.026036    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aff7a503-9cf4-4538-8bd5-da7ad324a6af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff7a503-9cf4-4538-8bd5-da7ad324a6af-maas-api-xks-ca\") pod \"aff7a503-9cf4-4538-8bd5-da7ad324a6af\" (UID: \"aff7a503-9cf4-4538-8bd5-da7ad324a6af\") "
Nov 03 08:28:43.026285 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.026148    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff7a503-9cf4-4538-8bd5-da7ad324a6af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff7a503-9cf4-4538-8bd5-da7ad324a6af-maas-api-openshift-ca\") pod \"aff7a503-9cf4-4538-8bd5-da7ad324a6af\" (UID: \"aff7a503-9cf4-4538-8bd5-da7ad324a6af\") "
Nov 03 08:28:43.026364 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.026332    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff7a503-9cf4-4538-8bd5-da7ad324a6af-kube-api-access-f9hwt\" (UniqueName: \"kubernetes.io/projected/aff7a503-9cf4-4538-8bd5-da7ad324a6af-kube-api-access-f9hwt\") pod \"aff7a503-9cf4-4538-8bd5-da7ad324a6af\" (UID: \"aff7a503-9cf4-4538-8bd5-da7ad324a6af\") "
Nov 03 08:28:43.026527 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.026501    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff7a503-9cf4-4538-8bd5-da7ad324a6af-maas-api-openshift-ca" pod "aff7a503-9cf4-4538-8bd5-da7ad324a6af" (UID: "aff7a503-9cf4-4538-8bd5-da7ad324a6af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:43.026802 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.026774    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aff7a503-9cf4-4538-8bd5-da7ad324a6af-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:43.028210 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.028182    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff7a503-9cf4-4538-8bd5-da7ad324a6af-maas-api-xks-ca" pod "aff7a503-9cf4-4538-8bd5-da7ad324a6af" (UID: "aff7a503-9cf4-4538-8bd5-da7ad324a6af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:43.028387 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.028361    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff7a503-9cf4-4538-8bd5-da7ad324a6af-kube-api-access-f9hwt" pod "aff7a503-9cf4-4538-8bd5-da7ad324a6af" (UID: "aff7a503-9cf4-4538-8bd5-da7ad324a6af"). InnerVolumeSpecName "kube-api-access-f9hwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:43.128281 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.128246    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9hwt\" (UniqueName: \"kubernetes.io/projected/aff7a503-9cf4-4538-8bd5-da7ad324a6af-kube-api-access-f9hwt\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:43.128281 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.128276    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aff7a503-9cf4-4538-8bd5-da7ad324a6af-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:43.618698 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.618658    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44"]
Nov 03 08:28:43.801852 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.801818    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm"]
Nov 03 08:28:43.802505 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.802483    5070 cpu_manager.go:414] "containerMap: removing container" podUID="aff7a503-9cf4-4538-8bd5-da7ad324a6af" containerName="revoke-keys"
Nov 03 08:28:43.802602 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.802508    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff7a503-9cf4-4538-8bd5-da7ad324a6af" containerName="revoke-keys"
Nov 03 08:28:43.802722 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.802704    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff7a503-9cf4-4538-8bd5-da7ad324a6af" containerName="revoke-keys"
Nov 03 08:28:43.806745 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.806711    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm"
Nov 03 08:28:43.825798 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.823934    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm"]
Nov 03 08:28:43.836821 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.836790    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfd9bcb3-lgxxm\" (UID: \"55536e9b-0a23-40c8-b9d5-0e9fe2a065c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm"
Nov 03 08:28:43.837000 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.836977    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4qpf\" (UniqueName: \"kubernetes.io/projected/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0-kube-api-access-z4qpf\") pod \"maas-api-revoke-sub-dfd9bcb3-lgxxm\" (UID: \"55536e9b-0a23-40c8-b9d5-0e9fe2a065c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm"
Nov 03 08:28:43.837070 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.837050    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfd9bcb3-lgxxm\" (UID: \"55536e9b-0a23-40c8-b9d5-0e9fe2a065c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm"
Nov 03 08:28:43.897817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.897267    5070 generic.go:356] "Generic (PLEG): container finished" podID="4fcd6530-2913-45a2-a36b-697f56d42909" containerID="7f5cb43858694e81fd3723a628ef723d32fff0f13de95c05b81def5ae0a0203e" exitCode=0
Nov 03 08:28:43.897817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.897392    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk" event={"ID":"4fcd6530-2913-45a2-a36b-697f56d42909","Type":"ContainerDied","Data":"7f5cb43858694e81fd3723a628ef723d32fff0f13de95c05b81def5ae0a0203e"}
Nov 03 08:28:43.899444 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.899419    5070 generic.go:356] "Generic (PLEG): container finished" podID="216f610f-a3c5-4967-949d-fbabca17ed1a" containerID="b6361dc0082e8ba4e47b63f2a172f96825a618ffec32652a95f141a96a591727" exitCode=0
Nov 03 08:28:43.899551 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.899522    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t" event={"ID":"216f610f-a3c5-4967-949d-fbabca17ed1a","Type":"ContainerDied","Data":"b6361dc0082e8ba4e47b63f2a172f96825a618ffec32652a95f141a96a591727"}
Nov 03 08:28:43.901299 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.901275    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr"
Nov 03 08:28:43.901393 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.901318    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr" event={"ID":"aff7a503-9cf4-4538-8bd5-da7ad324a6af","Type":"ContainerDied","Data":"05e2bb151477c956c0fe9e747ff0ea2b4c1d9ddef51b9f4e8956ea1856bae532"}
Nov 03 08:28:43.901393 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.901357    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05e2bb151477c956c0fe9e747ff0ea2b4c1d9ddef51b9f4e8956ea1856bae532"
Nov 03 08:28:43.937962 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.937916    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfd9bcb3-lgxxm\" (UID: \"55536e9b-0a23-40c8-b9d5-0e9fe2a065c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm"
Nov 03 08:28:43.938118 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.938058    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4qpf\" (UniqueName: \"kubernetes.io/projected/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0-kube-api-access-z4qpf\") pod \"maas-api-revoke-sub-dfd9bcb3-lgxxm\" (UID: \"55536e9b-0a23-40c8-b9d5-0e9fe2a065c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm"
Nov 03 08:28:43.938193 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.938127    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfd9bcb3-lgxxm\" (UID: \"55536e9b-0a23-40c8-b9d5-0e9fe2a065c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm"
Nov 03 08:28:43.940464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.940180    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfd9bcb3-lgxxm\" (UID: \"55536e9b-0a23-40c8-b9d5-0e9fe2a065c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm"
Nov 03 08:28:43.943430 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.943407    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfd9bcb3-lgxxm\" (UID: \"55536e9b-0a23-40c8-b9d5-0e9fe2a065c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm"
Nov 03 08:28:43.949444 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:43.949416    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4qpf\" (UniqueName: \"kubernetes.io/projected/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0-kube-api-access-z4qpf\") pod \"maas-api-revoke-sub-dfd9bcb3-lgxxm\" (UID: \"55536e9b-0a23-40c8-b9d5-0e9fe2a065c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm"
Nov 03 08:28:43.966687 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:28:43.966663    5070 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff7a503_9cf4_4538_8bd5_da7ad324a6af.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 08:28:44.019257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.019234    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk"
Nov 03 08:28:44.021729 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.021699    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t"
Nov 03 08:28:44.027519 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.027501    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq"]
Nov 03 08:28:44.027597 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.027527    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046740e4-g9hl5"]
Nov 03 08:28:44.028942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.028931    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef94017c-zz5fq"]
Nov 03 08:28:44.031445 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.031426    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046740e4-g9hl5"]
Nov 03 08:28:44.038899 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.038868    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fcd6530-2913-45a2-a36b-697f56d42909-kube-api-access-hhbmq\" (UniqueName: \"kubernetes.io/projected/4fcd6530-2913-45a2-a36b-697f56d42909-kube-api-access-hhbmq\") pod \"4fcd6530-2913-45a2-a36b-697f56d42909\" (UID: \"4fcd6530-2913-45a2-a36b-697f56d42909\") "
Nov 03 08:28:44.039019 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.038969    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/216f610f-a3c5-4967-949d-fbabca17ed1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/216f610f-a3c5-4967-949d-fbabca17ed1a-plugins-config-volume\") pod \"216f610f-a3c5-4967-949d-fbabca17ed1a\" (UID: \"216f610f-a3c5-4967-949d-fbabca17ed1a\") "
Nov 03 08:28:44.039019 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.039003    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/216f610f-a3c5-4967-949d-fbabca17ed1a-kube-api-access-phzbm\" (UniqueName: \"kubernetes.io/projected/216f610f-a3c5-4967-949d-fbabca17ed1a-kube-api-access-phzbm\") pod \"216f610f-a3c5-4967-949d-fbabca17ed1a\" (UID: \"216f610f-a3c5-4967-949d-fbabca17ed1a\") "
Nov 03 08:28:44.039179 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.039163    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fcd6530-2913-45a2-a36b-697f56d42909-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fcd6530-2913-45a2-a36b-697f56d42909-plugins-config-volume\") pod \"4fcd6530-2913-45a2-a36b-697f56d42909\" (UID: \"4fcd6530-2913-45a2-a36b-697f56d42909\") "
Nov 03 08:28:44.039622 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.039591    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fcd6530-2913-45a2-a36b-697f56d42909-plugins-config-volume" pod "4fcd6530-2913-45a2-a36b-697f56d42909" (UID: "4fcd6530-2913-45a2-a36b-697f56d42909"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:44.039729 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.039710    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/216f610f-a3c5-4967-949d-fbabca17ed1a-plugins-config-volume" pod "216f610f-a3c5-4967-949d-fbabca17ed1a" (UID: "216f610f-a3c5-4967-949d-fbabca17ed1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:44.041613 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.041584    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fcd6530-2913-45a2-a36b-697f56d42909-kube-api-access-hhbmq" pod "4fcd6530-2913-45a2-a36b-697f56d42909" (UID: "4fcd6530-2913-45a2-a36b-697f56d42909"). InnerVolumeSpecName "kube-api-access-hhbmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:44.042111 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.042095    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/216f610f-a3c5-4967-949d-fbabca17ed1a-kube-api-access-phzbm" pod "216f610f-a3c5-4967-949d-fbabca17ed1a" (UID: "216f610f-a3c5-4967-949d-fbabca17ed1a"). InnerVolumeSpecName "kube-api-access-phzbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:44.129566 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.129541    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm"
Nov 03 08:28:44.152801 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.141848    5070 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/216f610f-a3c5-4967-949d-fbabca17ed1a-plugins-config-volume\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:44.152801 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.141880    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phzbm\" (UniqueName: \"kubernetes.io/projected/216f610f-a3c5-4967-949d-fbabca17ed1a-kube-api-access-phzbm\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:44.152801 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.141907    5070 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fcd6530-2913-45a2-a36b-697f56d42909-plugins-config-volume\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:44.152801 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.141923    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhbmq\" (UniqueName: \"kubernetes.io/projected/4fcd6530-2913-45a2-a36b-697f56d42909-kube-api-access-hhbmq\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:44.410041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.410016    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm"]
Nov 03 08:28:44.411147 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:28:44.411125    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55536e9b_0a23_40c8_b9d5_0e9fe2a065c0.slice/crio-52373c4231f35d5869f80c73157fe7134e330b7c11ddbb2a4f5815e71782427f WatchSource:0}: Error finding container 52373c4231f35d5869f80c73157fe7134e330b7c11ddbb2a4f5815e71782427f: Status 404 returned error can't find the container with id 52373c4231f35d5869f80c73157fe7134e330b7c11ddbb2a4f5815e71782427f
Nov 03 08:28:44.910848 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.910805    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm" event={"ID":"55536e9b-0a23-40c8-b9d5-0e9fe2a065c0","Type":"ContainerStarted","Data":"0f18ccbd7c5bd67a07e2992fa79b1424b54ababd8a1d620cd68b6d1510541ade"}
Nov 03 08:28:44.911016 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.910866    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm" event={"ID":"55536e9b-0a23-40c8-b9d5-0e9fe2a065c0","Type":"ContainerStarted","Data":"52373c4231f35d5869f80c73157fe7134e330b7c11ddbb2a4f5815e71782427f"}
Nov 03 08:28:44.912270 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.912238    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk" event={"ID":"4fcd6530-2913-45a2-a36b-697f56d42909","Type":"ContainerDied","Data":"0dd54bb791cc1de06fdfcd9bbea955d0359adba87288b618f9e136d60680c0b7"}
Nov 03 08:28:44.912270 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.912264    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk"
Nov 03 08:28:44.912445 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.912294    5070 scope.go:122] "RemoveContainer" containerID="7f5cb43858694e81fd3723a628ef723d32fff0f13de95c05b81def5ae0a0203e"
Nov 03 08:28:44.913962 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.913938    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t" event={"ID":"216f610f-a3c5-4967-949d-fbabca17ed1a","Type":"ContainerDied","Data":"0bac3a35831b2fb6c9e83dd48429f4356096871a4ec9a9ba53070ab631971b1d"}
Nov 03 08:28:44.914103 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.913946    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t"
Nov 03 08:28:44.924456 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.924435    5070 scope.go:122] "RemoveContainer" containerID="b6361dc0082e8ba4e47b63f2a172f96825a618ffec32652a95f141a96a591727"
Nov 03 08:28:44.929824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.929739    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm" podStartSLOduration=1.929727904 podStartE2EDuration="1.929727904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:28:44.925568375 +0000 UTC m=+2123.040032306" watchObservedRunningTime="2026-11-03 08:28:44.929727904 +0000 UTC m=+2123.044191841"
Nov 03 08:28:44.940127 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.940105    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk"]
Nov 03 08:28:44.943616 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.943597    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d168e-6c8f45b799-9znfk"]
Nov 03 08:28:44.953514 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.953495    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t"]
Nov 03 08:28:44.956844 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:44.956826    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d168e-8698cff995-jrh4t"]
Nov 03 08:28:45.925569 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:45.925535    5070 generic.go:356] "Generic (PLEG): container finished" podID="55536e9b-0a23-40c8-b9d5-0e9fe2a065c0" containerID="0f18ccbd7c5bd67a07e2992fa79b1424b54ababd8a1d620cd68b6d1510541ade" exitCode=0
Nov 03 08:28:45.925943 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:45.925592    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm" event={"ID":"55536e9b-0a23-40c8-b9d5-0e9fe2a065c0","Type":"ContainerDied","Data":"0f18ccbd7c5bd67a07e2992fa79b1424b54ababd8a1d620cd68b6d1510541ade"}
Nov 03 08:28:46.024735 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:46.024706    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20d8ac2c-99c2-4125-ae50-cd90c6a78609" path="/var/lib/kubelet/pods/20d8ac2c-99c2-4125-ae50-cd90c6a78609/volumes"
Nov 03 08:28:46.025135 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:46.025120    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="216f610f-a3c5-4967-949d-fbabca17ed1a" path="/var/lib/kubelet/pods/216f610f-a3c5-4967-949d-fbabca17ed1a/volumes"
Nov 03 08:28:46.025492 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:46.025480    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4250e197-3658-4c5d-8130-35878b72827d" path="/var/lib/kubelet/pods/4250e197-3658-4c5d-8130-35878b72827d/volumes"
Nov 03 08:28:46.026151 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:46.026137    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fcd6530-2913-45a2-a36b-697f56d42909" path="/var/lib/kubelet/pods/4fcd6530-2913-45a2-a36b-697f56d42909/volumes"
Nov 03 08:28:47.107193 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.107170    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm"
Nov 03 08:28:47.171494 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.171463    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0-maas-api-openshift-ca\") pod \"55536e9b-0a23-40c8-b9d5-0e9fe2a065c0\" (UID: \"55536e9b-0a23-40c8-b9d5-0e9fe2a065c0\") "
Nov 03 08:28:47.171684 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.171504    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0-maas-api-xks-ca\") pod \"55536e9b-0a23-40c8-b9d5-0e9fe2a065c0\" (UID: \"55536e9b-0a23-40c8-b9d5-0e9fe2a065c0\") "
Nov 03 08:28:47.171684 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.171548    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0-kube-api-access-z4qpf\" (UniqueName: \"kubernetes.io/projected/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0-kube-api-access-z4qpf\") pod \"55536e9b-0a23-40c8-b9d5-0e9fe2a065c0\" (UID: \"55536e9b-0a23-40c8-b9d5-0e9fe2a065c0\") "
Nov 03 08:28:47.171910 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.171878    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0-maas-api-openshift-ca" pod "55536e9b-0a23-40c8-b9d5-0e9fe2a065c0" (UID: "55536e9b-0a23-40c8-b9d5-0e9fe2a065c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:28:47.172074 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.172057    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.173707 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.173672    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0-maas-api-xks-ca" pod "55536e9b-0a23-40c8-b9d5-0e9fe2a065c0" (UID: "55536e9b-0a23-40c8-b9d5-0e9fe2a065c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:47.173707 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.173697    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0-kube-api-access-z4qpf" pod "55536e9b-0a23-40c8-b9d5-0e9fe2a065c0" (UID: "55536e9b-0a23-40c8-b9d5-0e9fe2a065c0"). InnerVolumeSpecName "kube-api-access-z4qpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:47.273726 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.273642    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.273726 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.273671    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4qpf\" (UniqueName: \"kubernetes.io/projected/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0-kube-api-access-z4qpf\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.488514 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.488463    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main" containerID="cri-o://903c85d9d89bb11fe84693e4fbf2ac4e38b88fc9b1fbbacd35df488b83412824" gracePeriod=45
Nov 03 08:28:47.499464 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.499430    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" containerID="cri-o://0d350a5683831cadc84463292d1d7791cbc2733f865752dfb67ebf35ffba5735" gracePeriod=45
Nov 03 08:28:47.778183 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.777930    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:28:47.812255 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.812181    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:28:47.890725 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.890692    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-tmp-dir\") pod \"77e512c2-d43c-452c-86a3-7cf6bfea1558\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") "
Nov 03 08:28:47.891016 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.890772    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-tmp-dir\") pod \"381033e1-f38c-48ba-9bf2-45ba216df6fb\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") "
Nov 03 08:28:47.891016 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.890799    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-model-cache\") pod \"77e512c2-d43c-452c-86a3-7cf6bfea1558\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") "
Nov 03 08:28:47.891016 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.890838    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77e512c2-d43c-452c-86a3-7cf6bfea1558-tls-certs\" (UniqueName: \"kubernetes.io/secret/77e512c2-d43c-452c-86a3-7cf6bfea1558-tls-certs\") pod \"77e512c2-d43c-452c-86a3-7cf6bfea1558\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") "
Nov 03 08:28:47.891016 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.890915    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/381033e1-f38c-48ba-9bf2-45ba216df6fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/381033e1-f38c-48ba-9bf2-45ba216df6fb-tls-certs\") pod \"381033e1-f38c-48ba-9bf2-45ba216df6fb\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") "
Nov 03 08:28:47.891016 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.890949    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-dshm\") pod \"381033e1-f38c-48ba-9bf2-45ba216df6fb\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") "
Nov 03 08:28:47.891016 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.890993    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-home\" (UniqueName: \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-home\") pod \"381033e1-f38c-48ba-9bf2-45ba216df6fb\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") "
Nov 03 08:28:47.891375 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.891057    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/381033e1-f38c-48ba-9bf2-45ba216df6fb-kube-api-access-qmmdp\" (UniqueName: \"kubernetes.io/projected/381033e1-f38c-48ba-9bf2-45ba216df6fb-kube-api-access-qmmdp\") pod \"381033e1-f38c-48ba-9bf2-45ba216df6fb\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") "
Nov 03 08:28:47.891375 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.891092    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77e512c2-d43c-452c-86a3-7cf6bfea1558-kube-api-access-lssjk\" (UniqueName: \"kubernetes.io/projected/77e512c2-d43c-452c-86a3-7cf6bfea1558-kube-api-access-lssjk\") pod \"77e512c2-d43c-452c-86a3-7cf6bfea1558\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") "
Nov 03 08:28:47.891375 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.891125    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-home\" (UniqueName: \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-home\") pod \"77e512c2-d43c-452c-86a3-7cf6bfea1558\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") "
Nov 03 08:28:47.891375 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.891124    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-tmp-dir" pod "77e512c2-d43c-452c-86a3-7cf6bfea1558" (UID: "77e512c2-d43c-452c-86a3-7cf6bfea1558"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:47.891375 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.891139    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-model-cache" pod "77e512c2-d43c-452c-86a3-7cf6bfea1558" (UID: "77e512c2-d43c-452c-86a3-7cf6bfea1558"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:47.891375 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.891156    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-dshm\" (UniqueName: \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-dshm\") pod \"77e512c2-d43c-452c-86a3-7cf6bfea1558\" (UID: \"77e512c2-d43c-452c-86a3-7cf6bfea1558\") "
Nov 03 08:28:47.891375 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.891199    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-model-cache\") pod \"381033e1-f38c-48ba-9bf2-45ba216df6fb\" (UID: \"381033e1-f38c-48ba-9bf2-45ba216df6fb\") "
Nov 03 08:28:47.891805 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.891637    5070 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-tmp-dir\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.891805 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.891660    5070 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-model-cache\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.891805 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.891131    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-tmp-dir" pod "381033e1-f38c-48ba-9bf2-45ba216df6fb" (UID: "381033e1-f38c-48ba-9bf2-45ba216df6fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:47.891805 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.891616    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-home" pod "381033e1-f38c-48ba-9bf2-45ba216df6fb" (UID: "381033e1-f38c-48ba-9bf2-45ba216df6fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:47.891805 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.891734    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-home" pod "77e512c2-d43c-452c-86a3-7cf6bfea1558" (UID: "77e512c2-d43c-452c-86a3-7cf6bfea1558"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:47.892566 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.892542    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-model-cache" pod "381033e1-f38c-48ba-9bf2-45ba216df6fb" (UID: "381033e1-f38c-48ba-9bf2-45ba216df6fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:47.894178 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.894143    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-dshm" pod "77e512c2-d43c-452c-86a3-7cf6bfea1558" (UID: "77e512c2-d43c-452c-86a3-7cf6bfea1558"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:47.894178 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.894155    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77e512c2-d43c-452c-86a3-7cf6bfea1558-tls-certs" pod "77e512c2-d43c-452c-86a3-7cf6bfea1558" (UID: "77e512c2-d43c-452c-86a3-7cf6bfea1558"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:47.894354 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.894269    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381033e1-f38c-48ba-9bf2-45ba216df6fb-kube-api-access-qmmdp" pod "381033e1-f38c-48ba-9bf2-45ba216df6fb" (UID: "381033e1-f38c-48ba-9bf2-45ba216df6fb"). InnerVolumeSpecName "kube-api-access-qmmdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:47.894411 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.894344    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e512c2-d43c-452c-86a3-7cf6bfea1558-kube-api-access-lssjk" pod "77e512c2-d43c-452c-86a3-7cf6bfea1558" (UID: "77e512c2-d43c-452c-86a3-7cf6bfea1558"). InnerVolumeSpecName "kube-api-access-lssjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:47.894632 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.894613    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-dshm" pod "381033e1-f38c-48ba-9bf2-45ba216df6fb" (UID: "381033e1-f38c-48ba-9bf2-45ba216df6fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:47.895024 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.894986    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381033e1-f38c-48ba-9bf2-45ba216df6fb-tls-certs" pod "381033e1-f38c-48ba-9bf2-45ba216df6fb" (UID: "381033e1-f38c-48ba-9bf2-45ba216df6fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:47.946856 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.946825    5070 generic.go:356] "Generic (PLEG): container finished" podID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerID="0d350a5683831cadc84463292d1d7791cbc2733f865752dfb67ebf35ffba5735" exitCode=0
Nov 03 08:28:47.947041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.946870    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"
Nov 03 08:28:47.947041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.946904    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" event={"ID":"77e512c2-d43c-452c-86a3-7cf6bfea1558","Type":"ContainerDied","Data":"0d350a5683831cadc84463292d1d7791cbc2733f865752dfb67ebf35ffba5735"}
Nov 03 08:28:47.947041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.946950    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" event={"ID":"77e512c2-d43c-452c-86a3-7cf6bfea1558","Type":"ContainerDied","Data":"bc8fa99e6de6d8edcef8f9a8e2fb14e2b54d67f125a8655fd6cffa3e65ab04a8"}
Nov 03 08:28:47.947041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.946971    5070 scope.go:122] "RemoveContainer" containerID="0d350a5683831cadc84463292d1d7791cbc2733f865752dfb67ebf35ffba5735"
Nov 03 08:28:47.950126 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.950110    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm"
Nov 03 08:28:47.950190 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.950152    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm" event={"ID":"55536e9b-0a23-40c8-b9d5-0e9fe2a065c0","Type":"ContainerDied","Data":"52373c4231f35d5869f80c73157fe7134e330b7c11ddbb2a4f5815e71782427f"}
Nov 03 08:28:47.950245 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.950206    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52373c4231f35d5869f80c73157fe7134e330b7c11ddbb2a4f5815e71782427f"
Nov 03 08:28:47.951839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.951813    5070 generic.go:356] "Generic (PLEG): container finished" podID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerID="903c85d9d89bb11fe84693e4fbf2ac4e38b88fc9b1fbbacd35df488b83412824" exitCode=0
Nov 03 08:28:47.952040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.951862    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"
Nov 03 08:28:47.952040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.951917    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" event={"ID":"381033e1-f38c-48ba-9bf2-45ba216df6fb","Type":"ContainerDied","Data":"903c85d9d89bb11fe84693e4fbf2ac4e38b88fc9b1fbbacd35df488b83412824"}
Nov 03 08:28:47.952040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.951955    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc" event={"ID":"381033e1-f38c-48ba-9bf2-45ba216df6fb","Type":"ContainerDied","Data":"8be46b1b9934259d609f54d2b2f42a7af5af1cd4f77e7af1c9d31470ecc9a2a8"}
Nov 03 08:28:47.962899 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.962877    5070 scope.go:122] "RemoveContainer" containerID="0d350a5683831cadc84463292d1d7791cbc2733f865752dfb67ebf35ffba5735"
Nov 03 08:28:47.963202 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:28:47.963181    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d350a5683831cadc84463292d1d7791cbc2733f865752dfb67ebf35ffba5735\": container with ID starting with 0d350a5683831cadc84463292d1d7791cbc2733f865752dfb67ebf35ffba5735 not found: ID does not exist" containerID="0d350a5683831cadc84463292d1d7791cbc2733f865752dfb67ebf35ffba5735"
Nov 03 08:28:47.963294 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.963210    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d350a5683831cadc84463292d1d7791cbc2733f865752dfb67ebf35ffba5735"} err="failed to get container status \"0d350a5683831cadc84463292d1d7791cbc2733f865752dfb67ebf35ffba5735\": rpc error: code = NotFound desc = could not find container \"0d350a5683831cadc84463292d1d7791cbc2733f865752dfb67ebf35ffba5735\": container with ID starting with 0d350a5683831cadc84463292d1d7791cbc2733f865752dfb67ebf35ffba5735 not found: ID does not exist"
Nov 03 08:28:47.963294 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.963231    5070 scope.go:122] "RemoveContainer" containerID="903c85d9d89bb11fe84693e4fbf2ac4e38b88fc9b1fbbacd35df488b83412824"
Nov 03 08:28:47.973605 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.973587    5070 scope.go:122] "RemoveContainer" containerID="903c85d9d89bb11fe84693e4fbf2ac4e38b88fc9b1fbbacd35df488b83412824"
Nov 03 08:28:47.973916 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:28:47.973894    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"903c85d9d89bb11fe84693e4fbf2ac4e38b88fc9b1fbbacd35df488b83412824\": container with ID starting with 903c85d9d89bb11fe84693e4fbf2ac4e38b88fc9b1fbbacd35df488b83412824 not found: ID does not exist" containerID="903c85d9d89bb11fe84693e4fbf2ac4e38b88fc9b1fbbacd35df488b83412824"
Nov 03 08:28:47.973984 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.973924    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"903c85d9d89bb11fe84693e4fbf2ac4e38b88fc9b1fbbacd35df488b83412824"} err="failed to get container status \"903c85d9d89bb11fe84693e4fbf2ac4e38b88fc9b1fbbacd35df488b83412824\": rpc error: code = NotFound desc = could not find container \"903c85d9d89bb11fe84693e4fbf2ac4e38b88fc9b1fbbacd35df488b83412824\": container with ID starting with 903c85d9d89bb11fe84693e4fbf2ac4e38b88fc9b1fbbacd35df488b83412824 not found: ID does not exist"
Nov 03 08:28:47.992359 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.992338    5070 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-home\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.992446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.992362    5070 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77e512c2-d43c-452c-86a3-7cf6bfea1558-dshm\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.992446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.992375    5070 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-model-cache\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.992446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.992384    5070 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-tmp-dir\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.992446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.992394    5070 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e512c2-d43c-452c-86a3-7cf6bfea1558-tls-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.992446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.992401    5070 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/381033e1-f38c-48ba-9bf2-45ba216df6fb-tls-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.992446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.992410    5070 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-dshm\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.992446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.992419    5070 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/381033e1-f38c-48ba-9bf2-45ba216df6fb-home\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.992446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.992427    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmmdp\" (UniqueName: \"kubernetes.io/projected/381033e1-f38c-48ba-9bf2-45ba216df6fb-kube-api-access-qmmdp\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:47.992446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:47.992438    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lssjk\" (UniqueName: \"kubernetes.io/projected/77e512c2-d43c-452c-86a3-7cf6bfea1558-kube-api-access-lssjk\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:48.005139 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:48.005108    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"]
Nov 03 08:28:48.011740 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:48.011720    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm"]
Nov 03 08:28:48.039055 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:48.039017    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" path="/var/lib/kubelet/pods/77e512c2-d43c-452c-86a3-7cf6bfea1558/volumes"
Nov 03 08:28:48.039439 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:48.039418    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"]
Nov 03 08:28:48.039538 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:48.039454    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ffe7a8/premium-w5-ffe7a8-kserve-79db785d58-2skmc"]
Nov 03 08:28:48.187925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:48.187888    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9" podUID="917243d5-933a-4f61-9e76-72ec95f6a0dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:28:48.619232 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:48.619179    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44" podUID="56258879-0d2c-457c-a0d2-0e7c378bed50" containerName="payload-pre-processing" containerID="cri-o://286a57fd568b2648bb4dd3481b47fb5a2a051c7afc381e04413c0db92e6d2d73" gracePeriod=25
Nov 03 08:28:48.748553 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:48.748508    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ffe7a8/simulator-w5-ffe7a8-kserve-575bfc8b9f-276dm" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 08:28:50.024213 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:50.024178    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" path="/var/lib/kubelet/pods/381033e1-f38c-48ba-9bf2-45ba216df6fb/volumes"
Nov 03 08:28:51.022523 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:51.022493    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz"]
Nov 03 08:28:51.026060 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:51.026029    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq"]
Nov 03 08:28:51.029565 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:51.029543    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e09976b-8xqzz"]
Nov 03 08:28:51.032004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:51.031971    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e376881b-5w7cq"]
Nov 03 08:28:52.024067 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:52.024033    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27376302-17ef-4469-9b08-d882df27e7de" path="/var/lib/kubelet/pods/27376302-17ef-4469-9b08-d882df27e7de/volumes"
Nov 03 08:28:52.024484 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:52.024469    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7" path="/var/lib/kubelet/pods/6f0e4de6-c3e9-45f3-8542-91ed6f0f0bf7/volumes"
Nov 03 08:28:52.050484 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:28:52.050453    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:28:53.322384 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:53.322329    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" containerID="cri-o://b4ef020ec54f355d310476ff5f1bf4b13fcfb9f317b24ca9f693e078617c3daa" gracePeriod=45
Nov 03 08:28:53.494875 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:53.494728    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:28:53.587710 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:53.587658    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:28:53.655007 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:53.654973    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ae55a46-f433-42f5-b29b-dc0c51b87de4-tls-certs\" (UniqueName: \"kubernetes.io/secret/8ae55a46-f433-42f5-b29b-dc0c51b87de4-tls-certs\") pod \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") "
Nov 03 08:28:53.655177 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:53.655087    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-dshm\") pod \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") "
Nov 03 08:28:53.655177 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:53.655135    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-model-cache\") pod \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") "
Nov 03 08:28:53.655315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:53.655192    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-tmp-dir\") pod \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") "
Nov 03 08:28:53.655315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:53.655290    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-home\" (UniqueName: \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-home\") pod \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") "
Nov 03 08:28:53.655440 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:53.655333    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae55a46-f433-42f5-b29b-dc0c51b87de4-kube-api-access-mx4bt\" (UniqueName: \"kubernetes.io/projected/8ae55a46-f433-42f5-b29b-dc0c51b87de4-kube-api-access-mx4bt\") pod \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\" (UID: \"8ae55a46-f433-42f5-b29b-dc0c51b87de4\") "
Nov 03 08:28:53.656112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:53.656073    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-home" pod "8ae55a46-f433-42f5-b29b-dc0c51b87de4" (UID: "8ae55a46-f433-42f5-b29b-dc0c51b87de4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:53.656235 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:53.656157    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-tmp-dir" pod "8ae55a46-f433-42f5-b29b-dc0c51b87de4" (UID: "8ae55a46-f433-42f5-b29b-dc0c51b87de4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:53.656394 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:53.656361    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-model-cache" pod "8ae55a46-f433-42f5-b29b-dc0c51b87de4" (UID: "8ae55a46-f433-42f5-b29b-dc0c51b87de4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:53.658004 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:53.657961    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ae55a46-f433-42f5-b29b-dc0c51b87de4-tls-certs" pod "8ae55a46-f433-42f5-b29b-dc0c51b87de4" (UID: "8ae55a46-f433-42f5-b29b-dc0c51b87de4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:28:53.658293 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:53.658231    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae55a46-f433-42f5-b29b-dc0c51b87de4-kube-api-access-mx4bt" pod "8ae55a46-f433-42f5-b29b-dc0c51b87de4" (UID: "8ae55a46-f433-42f5-b29b-dc0c51b87de4"). InnerVolumeSpecName "kube-api-access-mx4bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:28:53.658492 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:53.658461    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-dshm" pod "8ae55a46-f433-42f5-b29b-dc0c51b87de4" (UID: "8ae55a46-f433-42f5-b29b-dc0c51b87de4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:28:53.756508 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:53.756479    5070 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-home\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:53.756508 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:53.756506    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx4bt\" (UniqueName: \"kubernetes.io/projected/8ae55a46-f433-42f5-b29b-dc0c51b87de4-kube-api-access-mx4bt\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:53.756680 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:53.756517    5070 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ae55a46-f433-42f5-b29b-dc0c51b87de4-tls-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:53.756680 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:53.756526    5070 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-dshm\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:53.756680 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:53.756536    5070 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-model-cache\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:53.756680 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:53.756544    5070 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ae55a46-f433-42f5-b29b-dc0c51b87de4-tmp-dir\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:28:54.002678 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:54.002643    5070 generic.go:356] "Generic (PLEG): container finished" podID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerID="b4ef020ec54f355d310476ff5f1bf4b13fcfb9f317b24ca9f693e078617c3daa" exitCode=0
Nov 03 08:28:54.002902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:54.002686    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"
Nov 03 08:28:54.002902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:54.002736    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" event={"ID":"8ae55a46-f433-42f5-b29b-dc0c51b87de4","Type":"ContainerDied","Data":"b4ef020ec54f355d310476ff5f1bf4b13fcfb9f317b24ca9f693e078617c3daa"}
Nov 03 08:28:54.002902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:54.002818    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" event={"ID":"8ae55a46-f433-42f5-b29b-dc0c51b87de4","Type":"ContainerDied","Data":"19b856821075d87252b8b7dd2ecb8e8a714af2689b48c98d744691c1c6781db6"}
Nov 03 08:28:54.002902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:54.002846    5070 scope.go:122] "RemoveContainer" containerID="b4ef020ec54f355d310476ff5f1bf4b13fcfb9f317b24ca9f693e078617c3daa"
Nov 03 08:28:54.016557 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:54.016539    5070 scope.go:122] "RemoveContainer" containerID="b4ef020ec54f355d310476ff5f1bf4b13fcfb9f317b24ca9f693e078617c3daa"
Nov 03 08:28:54.016860 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:28:54.016842    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4ef020ec54f355d310476ff5f1bf4b13fcfb9f317b24ca9f693e078617c3daa\": container with ID starting with b4ef020ec54f355d310476ff5f1bf4b13fcfb9f317b24ca9f693e078617c3daa not found: ID does not exist" containerID="b4ef020ec54f355d310476ff5f1bf4b13fcfb9f317b24ca9f693e078617c3daa"
Nov 03 08:28:54.016908 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:54.016868    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4ef020ec54f355d310476ff5f1bf4b13fcfb9f317b24ca9f693e078617c3daa"} err="failed to get container status \"b4ef020ec54f355d310476ff5f1bf4b13fcfb9f317b24ca9f693e078617c3daa\": rpc error: code = NotFound desc = could not find container \"b4ef020ec54f355d310476ff5f1bf4b13fcfb9f317b24ca9f693e078617c3daa\": container with ID starting with b4ef020ec54f355d310476ff5f1bf4b13fcfb9f317b24ca9f693e078617c3daa not found: ID does not exist"
Nov 03 08:28:54.025172 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:54.025154    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"]
Nov 03 08:28:54.029740 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:54.029721    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l"]
Nov 03 08:28:54.507874 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:54.507833    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6bc553/distinct-2-w2-6bc553-kserve-87ff5ddf6-qcb8l" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": context deadline exceeded"
Nov 03 08:28:54.919857 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:54.919823    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf"]
Nov 03 08:28:54.922443 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:54.922416    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ffe7a8-hk7zf"]
Nov 03 08:28:56.024156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:56.024120    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" path="/var/lib/kubelet/pods/8ae55a46-f433-42f5-b29b-dc0c51b87de4/volumes"
Nov 03 08:28:56.024538 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:56.024525    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e01352d1-5185-4b50-9868-d2b0b5e38890" path="/var/lib/kubelet/pods/e01352d1-5185-4b50-9868-d2b0b5e38890/volumes"
Nov 03 08:28:56.140940 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:56.140910    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9"]
Nov 03 08:28:56.145152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:56.145125    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-6bc553-6txc9"]
Nov 03 08:28:57.181989 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:57.181952    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44" podUID="56258879-0d2c-457c-a0d2-0e7c378bed50" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:28:58.024226 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:58.024196    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51c92f1c-4491-4136-9800-e0475139651e" path="/var/lib/kubelet/pods/51c92f1c-4491-4136-9800-e0475139651e/volumes"
Nov 03 08:28:58.187508 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:28:58.187475    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9" podUID="917243d5-933a-4f61-9e76-72ec95f6a0dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:29:01.643740 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:01.643699    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"]
Nov 03 08:29:01.644260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:01.644012    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv" podUID="e724a396-646c-42bb-9b59-d7ff618e07f8" containerName="istio-proxy" containerID="cri-o://59fbd2ff87bf8abbb5f680d7631cc95bfabb11fe11c6a344dd0c4b87f587df33" gracePeriod=30
Nov 03 08:29:03.029710 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:03.029657    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9"
Nov 03 08:29:03.090276 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:03.090247    5070 generic.go:356] "Generic (PLEG): container finished" podID="917243d5-933a-4f61-9e76-72ec95f6a0dd" containerID="311bea97dce6e955643a8cacfee6d4af6170e9aa483d13c65db7926ed4bd1e30" exitCode=0
Nov 03 08:29:03.090449 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:03.090288    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9"
Nov 03 08:29:03.090449 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:03.090329    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9" event={"ID":"917243d5-933a-4f61-9e76-72ec95f6a0dd","Type":"ContainerDied","Data":"311bea97dce6e955643a8cacfee6d4af6170e9aa483d13c65db7926ed4bd1e30"}
Nov 03 08:29:03.090449 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:03.090387    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9" event={"ID":"917243d5-933a-4f61-9e76-72ec95f6a0dd","Type":"ContainerDied","Data":"3f15a0b850a9e3084be6c54735290d40b7c3c81e19a90304249fb17d35ed973a"}
Nov 03 08:29:03.090449 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:03.090419    5070 scope.go:122] "RemoveContainer" containerID="311bea97dce6e955643a8cacfee6d4af6170e9aa483d13c65db7926ed4bd1e30"
Nov 03 08:29:03.101327 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:03.101305    5070 scope.go:122] "RemoveContainer" containerID="311bea97dce6e955643a8cacfee6d4af6170e9aa483d13c65db7926ed4bd1e30"
Nov 03 08:29:03.101603 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:29:03.101581    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"311bea97dce6e955643a8cacfee6d4af6170e9aa483d13c65db7926ed4bd1e30\": container with ID starting with 311bea97dce6e955643a8cacfee6d4af6170e9aa483d13c65db7926ed4bd1e30 not found: ID does not exist" containerID="311bea97dce6e955643a8cacfee6d4af6170e9aa483d13c65db7926ed4bd1e30"
Nov 03 08:29:03.101677 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:03.101611    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"311bea97dce6e955643a8cacfee6d4af6170e9aa483d13c65db7926ed4bd1e30"} err="failed to get container status \"311bea97dce6e955643a8cacfee6d4af6170e9aa483d13c65db7926ed4bd1e30\": rpc error: code = NotFound desc = could not find container \"311bea97dce6e955643a8cacfee6d4af6170e9aa483d13c65db7926ed4bd1e30\": container with ID starting with 311bea97dce6e955643a8cacfee6d4af6170e9aa483d13c65db7926ed4bd1e30 not found: ID does not exist"
Nov 03 08:29:03.146003 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:03.145980    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/917243d5-933a-4f61-9e76-72ec95f6a0dd-kube-api-access-w68bt\" (UniqueName: \"kubernetes.io/projected/917243d5-933a-4f61-9e76-72ec95f6a0dd-kube-api-access-w68bt\") pod \"917243d5-933a-4f61-9e76-72ec95f6a0dd\" (UID: \"917243d5-933a-4f61-9e76-72ec95f6a0dd\") "
Nov 03 08:29:03.146131 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:03.146023    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/917243d5-933a-4f61-9e76-72ec95f6a0dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/917243d5-933a-4f61-9e76-72ec95f6a0dd-plugins-config-volume\") pod \"917243d5-933a-4f61-9e76-72ec95f6a0dd\" (UID: \"917243d5-933a-4f61-9e76-72ec95f6a0dd\") "
Nov 03 08:29:03.146548 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:03.146481    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/917243d5-933a-4f61-9e76-72ec95f6a0dd-plugins-config-volume" pod "917243d5-933a-4f61-9e76-72ec95f6a0dd" (UID: "917243d5-933a-4f61-9e76-72ec95f6a0dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:29:03.149076 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:03.149047    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/917243d5-933a-4f61-9e76-72ec95f6a0dd-kube-api-access-w68bt" pod "917243d5-933a-4f61-9e76-72ec95f6a0dd" (UID: "917243d5-933a-4f61-9e76-72ec95f6a0dd"). InnerVolumeSpecName "kube-api-access-w68bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:29:03.186882 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:03.186850    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"]
Nov 03 08:29:03.187297 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:03.187270    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy" containerID="cri-o://852a632cb1812b8fba2b1ae708c51a02a9a8a0f2fe587cded79faafd9309df23" gracePeriod=30
Nov 03 08:29:03.248045 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:03.248014    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w68bt\" (UniqueName: \"kubernetes.io/projected/917243d5-933a-4f61-9e76-72ec95f6a0dd-kube-api-access-w68bt\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:03.248045 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:03.248042    5070 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/917243d5-933a-4f61-9e76-72ec95f6a0dd-plugins-config-volume\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:03.417280 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:03.417249    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9"]
Nov 03 08:29:03.420522 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:03.420493    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ffe7a8-7875c6f495-n4vp9"]
Nov 03 08:29:04.024681 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:04.024650    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="917243d5-933a-4f61-9e76-72ec95f6a0dd" path="/var/lib/kubelet/pods/917243d5-933a-4f61-9e76-72ec95f6a0dd/volumes"
Nov 03 08:29:04.490873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:04.490843    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:29:06.306821 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:06.306784    5070 patch_prober.go:29] interesting pod/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:29:06.307213 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:06.306848    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:29:06.935958 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:06.935897    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:29:07.080954 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.080925    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-credential-socket\") pod \"e724a396-646c-42bb-9b59-d7ff618e07f8\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") "
Nov 03 08:29:07.081189 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.080970    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-envoy\") pod \"e724a396-646c-42bb-9b59-d7ff618e07f8\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") "
Nov 03 08:29:07.081189 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.080997    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-data\") pod \"e724a396-646c-42bb-9b59-d7ff618e07f8\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") "
Nov 03 08:29:07.081189 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.081023    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e724a396-646c-42bb-9b59-d7ff618e07f8-kube-api-access-9t5ff\" (UniqueName: \"kubernetes.io/projected/e724a396-646c-42bb-9b59-d7ff618e07f8-kube-api-access-9t5ff\") pod \"e724a396-646c-42bb-9b59-d7ff618e07f8\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") "
Nov 03 08:29:07.081189 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.081073    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-workload-certs\") pod \"e724a396-646c-42bb-9b59-d7ff618e07f8\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") "
Nov 03 08:29:07.081189 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.081115    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e724a396-646c-42bb-9b59-d7ff618e07f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e724a396-646c-42bb-9b59-d7ff618e07f8-istiod-ca-cert\") pod \"e724a396-646c-42bb-9b59-d7ff618e07f8\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") "
Nov 03 08:29:07.081189 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.081153    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-podinfo\") pod \"e724a396-646c-42bb-9b59-d7ff618e07f8\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") "
Nov 03 08:29:07.081189 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.081175    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-token\" (UniqueName: \"kubernetes.io/projected/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-token\") pod \"e724a396-646c-42bb-9b59-d7ff618e07f8\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") "
Nov 03 08:29:07.081591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.081223    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-workload-socket\") pod \"e724a396-646c-42bb-9b59-d7ff618e07f8\" (UID: \"e724a396-646c-42bb-9b59-d7ff618e07f8\") "
Nov 03 08:29:07.081591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.081320    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-credential-socket" pod "e724a396-646c-42bb-9b59-d7ff618e07f8" (UID: "e724a396-646c-42bb-9b59-d7ff618e07f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:29:07.081591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.081421    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-workload-certs" pod "e724a396-646c-42bb-9b59-d7ff618e07f8" (UID: "e724a396-646c-42bb-9b59-d7ff618e07f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:29:07.081591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.081535    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-data" pod "e724a396-646c-42bb-9b59-d7ff618e07f8" (UID: "e724a396-646c-42bb-9b59-d7ff618e07f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:29:07.081833 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.081588    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e724a396-646c-42bb-9b59-d7ff618e07f8-istiod-ca-cert" pod "e724a396-646c-42bb-9b59-d7ff618e07f8" (UID: "e724a396-646c-42bb-9b59-d7ff618e07f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:29:07.081833 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.081606    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:07.081833 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.081627    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:07.081833 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.081773    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-workload-socket" pod "e724a396-646c-42bb-9b59-d7ff618e07f8" (UID: "e724a396-646c-42bb-9b59-d7ff618e07f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:29:07.083360 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.083322    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e724a396-646c-42bb-9b59-d7ff618e07f8-kube-api-access-9t5ff" pod "e724a396-646c-42bb-9b59-d7ff618e07f8" (UID: "e724a396-646c-42bb-9b59-d7ff618e07f8"). InnerVolumeSpecName "kube-api-access-9t5ff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:29:07.083489 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.083383    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-podinfo" pod "e724a396-646c-42bb-9b59-d7ff618e07f8" (UID: "e724a396-646c-42bb-9b59-d7ff618e07f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:29:07.083664 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.083644    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-envoy" pod "e724a396-646c-42bb-9b59-d7ff618e07f8" (UID: "e724a396-646c-42bb-9b59-d7ff618e07f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:29:07.083775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.083695    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-token" pod "e724a396-646c-42bb-9b59-d7ff618e07f8" (UID: "e724a396-646c-42bb-9b59-d7ff618e07f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:29:07.123050 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.123023    5070 generic.go:356] "Generic (PLEG): container finished" podID="e724a396-646c-42bb-9b59-d7ff618e07f8" containerID="59fbd2ff87bf8abbb5f680d7631cc95bfabb11fe11c6a344dd0c4b87f587df33" exitCode=0
Nov 03 08:29:07.123149 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.123076    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"
Nov 03 08:29:07.123149 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.123108    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv" event={"ID":"e724a396-646c-42bb-9b59-d7ff618e07f8","Type":"ContainerDied","Data":"59fbd2ff87bf8abbb5f680d7631cc95bfabb11fe11c6a344dd0c4b87f587df33"}
Nov 03 08:29:07.123227 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.123163    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv" event={"ID":"e724a396-646c-42bb-9b59-d7ff618e07f8","Type":"ContainerDied","Data":"79f83c0c38ef05a99da3f3b313953ce6a8f0a1ffeb4ef8b4d6d624558f5e8320"}
Nov 03 08:29:07.123227 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.123183    5070 scope.go:122] "RemoveContainer" containerID="59fbd2ff87bf8abbb5f680d7631cc95bfabb11fe11c6a344dd0c4b87f587df33"
Nov 03 08:29:07.140426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.140400    5070 scope.go:122] "RemoveContainer" containerID="59fbd2ff87bf8abbb5f680d7631cc95bfabb11fe11c6a344dd0c4b87f587df33"
Nov 03 08:29:07.140982 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:29:07.140958    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59fbd2ff87bf8abbb5f680d7631cc95bfabb11fe11c6a344dd0c4b87f587df33\": container with ID starting with 59fbd2ff87bf8abbb5f680d7631cc95bfabb11fe11c6a344dd0c4b87f587df33 not found: ID does not exist" containerID="59fbd2ff87bf8abbb5f680d7631cc95bfabb11fe11c6a344dd0c4b87f587df33"
Nov 03 08:29:07.141098 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.140995    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59fbd2ff87bf8abbb5f680d7631cc95bfabb11fe11c6a344dd0c4b87f587df33"} err="failed to get container status \"59fbd2ff87bf8abbb5f680d7631cc95bfabb11fe11c6a344dd0c4b87f587df33\": rpc error: code = NotFound desc = could not find container \"59fbd2ff87bf8abbb5f680d7631cc95bfabb11fe11c6a344dd0c4b87f587df33\": container with ID starting with 59fbd2ff87bf8abbb5f680d7631cc95bfabb11fe11c6a344dd0c4b87f587df33 not found: ID does not exist"
Nov 03 08:29:07.145677 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.145655    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"]
Nov 03 08:29:07.148294 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.148271    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ffe7a8-openshift-default-59c4db678f-wc2fv"]
Nov 03 08:29:07.181627 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.181605    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44" podUID="56258879-0d2c-457c-a0d2-0e7c378bed50" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 08:29:07.182156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.182134    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t5ff\" (UniqueName: \"kubernetes.io/projected/e724a396-646c-42bb-9b59-d7ff618e07f8-kube-api-access-9t5ff\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:07.182256 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.182159    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e724a396-646c-42bb-9b59-d7ff618e07f8-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:07.182256 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.182174    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:07.182256 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.182187    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:07.182256 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.182201    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:07.182256 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.182215    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:07.182256 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:07.182228    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e724a396-646c-42bb-9b59-d7ff618e07f8-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:08.025385 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.025350    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e724a396-646c-42bb-9b59-d7ff618e07f8" path="/var/lib/kubelet/pods/e724a396-646c-42bb-9b59-d7ff618e07f8/volumes"
Nov 03 08:29:08.454084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.454054    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:29:08.595748 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.595660    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0822706-f191-47c7-8c87-e373872590e3-kube-api-access-fsx2t\" (UniqueName: \"kubernetes.io/projected/c0822706-f191-47c7-8c87-e373872590e3-kube-api-access-fsx2t\") pod \"c0822706-f191-47c7-8c87-e373872590e3\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") "
Nov 03 08:29:08.595928 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.595747    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0822706-f191-47c7-8c87-e373872590e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0822706-f191-47c7-8c87-e373872590e3-istio-podinfo\") pod \"c0822706-f191-47c7-8c87-e373872590e3\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") "
Nov 03 08:29:08.595928 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.595819    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0822706-f191-47c7-8c87-e373872590e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0822706-f191-47c7-8c87-e373872590e3-istiod-ca-cert\") pod \"c0822706-f191-47c7-8c87-e373872590e3\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") "
Nov 03 08:29:08.595928 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.595855    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-credential-socket\") pod \"c0822706-f191-47c7-8c87-e373872590e3\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") "
Nov 03 08:29:08.595928 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.595914    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-istio-data\") pod \"c0822706-f191-47c7-8c87-e373872590e3\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") "
Nov 03 08:29:08.596178 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.595949    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-workload-certs\") pod \"c0822706-f191-47c7-8c87-e373872590e3\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") "
Nov 03 08:29:08.596178 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.595982    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0822706-f191-47c7-8c87-e373872590e3-istio-token\" (UniqueName: \"kubernetes.io/projected/c0822706-f191-47c7-8c87-e373872590e3-istio-token\") pod \"c0822706-f191-47c7-8c87-e373872590e3\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") "
Nov 03 08:29:08.596178 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.596037    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-istio-envoy\") pod \"c0822706-f191-47c7-8c87-e373872590e3\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") "
Nov 03 08:29:08.596178 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.596121    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-workload-socket\") pod \"c0822706-f191-47c7-8c87-e373872590e3\" (UID: \"c0822706-f191-47c7-8c87-e373872590e3\") "
Nov 03 08:29:08.596389 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.596252    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0822706-f191-47c7-8c87-e373872590e3-istiod-ca-cert" pod "c0822706-f191-47c7-8c87-e373872590e3" (UID: "c0822706-f191-47c7-8c87-e373872590e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:29:08.596547 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.596523    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0822706-f191-47c7-8c87-e373872590e3-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:08.596836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.596810    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-workload-socket" pod "c0822706-f191-47c7-8c87-e373872590e3" (UID: "c0822706-f191-47c7-8c87-e373872590e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:29:08.596938 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.596828    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-workload-certs" pod "c0822706-f191-47c7-8c87-e373872590e3" (UID: "c0822706-f191-47c7-8c87-e373872590e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:29:08.596938 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.596898    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-istio-data" pod "c0822706-f191-47c7-8c87-e373872590e3" (UID: "c0822706-f191-47c7-8c87-e373872590e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:29:08.597070 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.596946    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-credential-socket" pod "c0822706-f191-47c7-8c87-e373872590e3" (UID: "c0822706-f191-47c7-8c87-e373872590e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:29:08.598604 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.598573    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0822706-f191-47c7-8c87-e373872590e3-istio-podinfo" pod "c0822706-f191-47c7-8c87-e373872590e3" (UID: "c0822706-f191-47c7-8c87-e373872590e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:29:08.598723 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.598603    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-istio-envoy" pod "c0822706-f191-47c7-8c87-e373872590e3" (UID: "c0822706-f191-47c7-8c87-e373872590e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:29:08.598723 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.598615    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0822706-f191-47c7-8c87-e373872590e3-kube-api-access-fsx2t" pod "c0822706-f191-47c7-8c87-e373872590e3" (UID: "c0822706-f191-47c7-8c87-e373872590e3"). InnerVolumeSpecName "kube-api-access-fsx2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:29:08.598916 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.598896    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0822706-f191-47c7-8c87-e373872590e3-istio-token" pod "c0822706-f191-47c7-8c87-e373872590e3" (UID: "c0822706-f191-47c7-8c87-e373872590e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:29:08.697723 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.697683    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:08.697723 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.697723    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:08.698008 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.697740    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsx2t\" (UniqueName: \"kubernetes.io/projected/c0822706-f191-47c7-8c87-e373872590e3-kube-api-access-fsx2t\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:08.698008 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.697772    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0822706-f191-47c7-8c87-e373872590e3-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:08.698008 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.697786    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:08.698008 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.697799    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:08.698008 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.697811    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0822706-f191-47c7-8c87-e373872590e3-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:08.698008 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:08.697825    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0822706-f191-47c7-8c87-e373872590e3-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:09.017979 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.017937    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44"
Nov 03 08:29:09.109459 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.109423    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56258879-0d2c-457c-a0d2-0e7c378bed50-kube-api-access-d84pm\" (UniqueName: \"kubernetes.io/projected/56258879-0d2c-457c-a0d2-0e7c378bed50-kube-api-access-d84pm\") pod \"56258879-0d2c-457c-a0d2-0e7c378bed50\" (UID: \"56258879-0d2c-457c-a0d2-0e7c378bed50\") "
Nov 03 08:29:09.109960 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.109605    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56258879-0d2c-457c-a0d2-0e7c378bed50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56258879-0d2c-457c-a0d2-0e7c378bed50-plugins-config-volume\") pod \"56258879-0d2c-457c-a0d2-0e7c378bed50\" (UID: \"56258879-0d2c-457c-a0d2-0e7c378bed50\") "
Nov 03 08:29:09.111099 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.111072    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56258879-0d2c-457c-a0d2-0e7c378bed50-plugins-config-volume" pod "56258879-0d2c-457c-a0d2-0e7c378bed50" (UID: "56258879-0d2c-457c-a0d2-0e7c378bed50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:29:09.113137 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.113106    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56258879-0d2c-457c-a0d2-0e7c378bed50-kube-api-access-d84pm" pod "56258879-0d2c-457c-a0d2-0e7c378bed50" (UID: "56258879-0d2c-457c-a0d2-0e7c378bed50"). InnerVolumeSpecName "kube-api-access-d84pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:29:09.142203 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.142180    5070 generic.go:356] "Generic (PLEG): container finished" podID="56258879-0d2c-457c-a0d2-0e7c378bed50" containerID="286a57fd568b2648bb4dd3481b47fb5a2a051c7afc381e04413c0db92e6d2d73" exitCode=0
Nov 03 08:29:09.142307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.142224    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44"
Nov 03 08:29:09.142307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.142259    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44" event={"ID":"56258879-0d2c-457c-a0d2-0e7c378bed50","Type":"ContainerDied","Data":"286a57fd568b2648bb4dd3481b47fb5a2a051c7afc381e04413c0db92e6d2d73"}
Nov 03 08:29:09.142423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.142314    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44" event={"ID":"56258879-0d2c-457c-a0d2-0e7c378bed50","Type":"ContainerDied","Data":"21ebfdd1ab12eeb5a4e922df261e64ed44a9fccffa455feb48c04fd22529d643"}
Nov 03 08:29:09.142423 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.142346    5070 scope.go:122] "RemoveContainer" containerID="286a57fd568b2648bb4dd3481b47fb5a2a051c7afc381e04413c0db92e6d2d73"
Nov 03 08:29:09.143917 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.143669    5070 generic.go:356] "Generic (PLEG): container finished" podID="c0822706-f191-47c7-8c87-e373872590e3" containerID="852a632cb1812b8fba2b1ae708c51a02a9a8a0f2fe587cded79faafd9309df23" exitCode=0
Nov 03 08:29:09.143917 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.143706    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"
Nov 03 08:29:09.143917 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.143745    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" event={"ID":"c0822706-f191-47c7-8c87-e373872590e3","Type":"ContainerDied","Data":"852a632cb1812b8fba2b1ae708c51a02a9a8a0f2fe587cded79faafd9309df23"}
Nov 03 08:29:09.143917 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.143811    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg" event={"ID":"c0822706-f191-47c7-8c87-e373872590e3","Type":"ContainerDied","Data":"a42525886e4fb381e03d003883a5074d19b6840aa9727cd623878f3683a25563"}
Nov 03 08:29:09.174478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.173944    5070 scope.go:122] "RemoveContainer" containerID="286a57fd568b2648bb4dd3481b47fb5a2a051c7afc381e04413c0db92e6d2d73"
Nov 03 08:29:09.174478 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:29:09.174375    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"286a57fd568b2648bb4dd3481b47fb5a2a051c7afc381e04413c0db92e6d2d73\": container with ID starting with 286a57fd568b2648bb4dd3481b47fb5a2a051c7afc381e04413c0db92e6d2d73 not found: ID does not exist" containerID="286a57fd568b2648bb4dd3481b47fb5a2a051c7afc381e04413c0db92e6d2d73"
Nov 03 08:29:09.174478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.174412    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"286a57fd568b2648bb4dd3481b47fb5a2a051c7afc381e04413c0db92e6d2d73"} err="failed to get container status \"286a57fd568b2648bb4dd3481b47fb5a2a051c7afc381e04413c0db92e6d2d73\": rpc error: code = NotFound desc = could not find container \"286a57fd568b2648bb4dd3481b47fb5a2a051c7afc381e04413c0db92e6d2d73\": container with ID starting with 286a57fd568b2648bb4dd3481b47fb5a2a051c7afc381e04413c0db92e6d2d73 not found: ID does not exist"
Nov 03 08:29:09.174478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.174440    5070 scope.go:122] "RemoveContainer" containerID="852a632cb1812b8fba2b1ae708c51a02a9a8a0f2fe587cded79faafd9309df23"
Nov 03 08:29:09.202460 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.202420    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"]
Nov 03 08:29:09.202864 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.202845    5070 scope.go:122] "RemoveContainer" containerID="852a632cb1812b8fba2b1ae708c51a02a9a8a0f2fe587cded79faafd9309df23"
Nov 03 08:29:09.203364 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:29:09.203333    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"852a632cb1812b8fba2b1ae708c51a02a9a8a0f2fe587cded79faafd9309df23\": container with ID starting with 852a632cb1812b8fba2b1ae708c51a02a9a8a0f2fe587cded79faafd9309df23 not found: ID does not exist" containerID="852a632cb1812b8fba2b1ae708c51a02a9a8a0f2fe587cded79faafd9309df23"
Nov 03 08:29:09.203539 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.203510    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"852a632cb1812b8fba2b1ae708c51a02a9a8a0f2fe587cded79faafd9309df23"} err="failed to get container status \"852a632cb1812b8fba2b1ae708c51a02a9a8a0f2fe587cded79faafd9309df23\": rpc error: code = NotFound desc = could not find container \"852a632cb1812b8fba2b1ae708c51a02a9a8a0f2fe587cded79faafd9309df23\": container with ID starting with 852a632cb1812b8fba2b1ae708c51a02a9a8a0f2fe587cded79faafd9309df23 not found: ID does not exist"
Nov 03 08:29:09.207555 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.207537    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6bc553-openshift-default-7f97fc7f9c-v6dqg"]
Nov 03 08:29:09.211343 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.211318    5070 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56258879-0d2c-457c-a0d2-0e7c378bed50-plugins-config-volume\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:09.211563 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.211526    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d84pm\" (UniqueName: \"kubernetes.io/projected/56258879-0d2c-457c-a0d2-0e7c378bed50-kube-api-access-d84pm\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:29:09.214467 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.214448    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44"]
Nov 03 08:29:09.218324 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:09.218299    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6bc553-67fcc4498c-4nv44"]
Nov 03 08:29:10.024976 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:10.024945    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56258879-0d2c-457c-a0d2-0e7c378bed50" path="/var/lib/kubelet/pods/56258879-0d2c-457c-a0d2-0e7c378bed50/volumes"
Nov 03 08:29:10.025384 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:10.025371    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0822706-f191-47c7-8c87-e373872590e3" path="/var/lib/kubelet/pods/c0822706-f191-47c7-8c87-e373872590e3/volumes"
Nov 03 08:29:14.628029 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:14.627992    5070 scope.go:122] "RemoveContainer" containerID="6da8397c627411c72a29ac5d413c4d8f89a199960ca3faeb2eed0e382e8655d4"
Nov 03 08:29:14.649946 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:14.649927    5070 scope.go:122] "RemoveContainer" containerID="401859e7ded8ab3ca1cd8bf947e33c6c596078f905eb2dbcebe14c491e02dc4c"
Nov 03 08:29:14.662396 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:14.662371    5070 scope.go:122] "RemoveContainer" containerID="621bd062dbfa2978c0edf06adaac5a1a0a4cba518c3028e4fd96698d60662f1e"
Nov 03 08:29:14.677925 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:14.677901    5070 scope.go:122] "RemoveContainer" containerID="83f57ad4dcc4607eea144ad3afb0ce83ad199c5b3db06de70697a3eb54913554"
Nov 03 08:29:14.687822 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:14.687730    5070 scope.go:122] "RemoveContainer" containerID="c0b057e62081994633a8b23d17c728488140db06194115ecb2facc7b4fbfd23f"
Nov 03 08:29:14.697347 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:14.697330    5070 scope.go:122] "RemoveContainer" containerID="cd909319e40c089e64bfd448294f742abda0f3b7e5cca03cda81489c6f2cbc85"
Nov 03 08:29:14.707941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:14.707914    5070 scope.go:122] "RemoveContainer" containerID="14ea11090aced2a25c739bcb463a9014561e8b797d8eba04ec2d558d6d30119f"
Nov 03 08:29:22.051246 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:29:22.051211    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:29:23.943715 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.943677    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"]
Nov 03 08:29:23.944386 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944365    5070 cpu_manager.go:414] "containerMap: removing container" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main"
Nov 03 08:29:23.944427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944391    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main"
Nov 03 08:29:23.944427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944414    5070 cpu_manager.go:414] "containerMap: removing container" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main"
Nov 03 08:29:23.944427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944423    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main"
Nov 03 08:29:23.944530 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944462    5070 cpu_manager.go:414] "containerMap: removing container" podUID="e724a396-646c-42bb-9b59-d7ff618e07f8" containerName="istio-proxy"
Nov 03 08:29:23.944530 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944469    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e724a396-646c-42bb-9b59-d7ff618e07f8" containerName="istio-proxy"
Nov 03 08:29:23.944530 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944487    5070 cpu_manager.go:414] "containerMap: removing container" podUID="4fcd6530-2913-45a2-a36b-697f56d42909" containerName="payload-pre-processing"
Nov 03 08:29:23.944530 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944495    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fcd6530-2913-45a2-a36b-697f56d42909" containerName="payload-pre-processing"
Nov 03 08:29:23.944530 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944512    5070 cpu_manager.go:414] "containerMap: removing container" podUID="56258879-0d2c-457c-a0d2-0e7c378bed50" containerName="payload-pre-processing"
Nov 03 08:29:23.944530 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944519    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56258879-0d2c-457c-a0d2-0e7c378bed50" containerName="payload-pre-processing"
Nov 03 08:29:23.944530 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944530    5070 cpu_manager.go:414] "containerMap: removing container" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy"
Nov 03 08:29:23.944774 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944537    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy"
Nov 03 08:29:23.944774 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944549    5070 cpu_manager.go:414] "containerMap: removing container" podUID="216f610f-a3c5-4967-949d-fbabca17ed1a" containerName="payload-processing"
Nov 03 08:29:23.944774 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944557    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="216f610f-a3c5-4967-949d-fbabca17ed1a" containerName="payload-processing"
Nov 03 08:29:23.944774 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944568    5070 cpu_manager.go:414] "containerMap: removing container" podUID="55536e9b-0a23-40c8-b9d5-0e9fe2a065c0" containerName="revoke-keys"
Nov 03 08:29:23.944774 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944576    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55536e9b-0a23-40c8-b9d5-0e9fe2a065c0" containerName="revoke-keys"
Nov 03 08:29:23.944774 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944593    5070 cpu_manager.go:414] "containerMap: removing container" podUID="917243d5-933a-4f61-9e76-72ec95f6a0dd" containerName="payload-processing"
Nov 03 08:29:23.944774 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944601    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="917243d5-933a-4f61-9e76-72ec95f6a0dd" containerName="payload-processing"
Nov 03 08:29:23.944774 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944611    5070 cpu_manager.go:414] "containerMap: removing container" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main"
Nov 03 08:29:23.944774 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944618    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main"
Nov 03 08:29:23.944774 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944773    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="917243d5-933a-4f61-9e76-72ec95f6a0dd" containerName="payload-processing"
Nov 03 08:29:23.945084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944786    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae55a46-f433-42f5-b29b-dc0c51b87de4" containerName="main"
Nov 03 08:29:23.945084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944803    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="77e512c2-d43c-452c-86a3-7cf6bfea1558" containerName="main"
Nov 03 08:29:23.945084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944817    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="381033e1-f38c-48ba-9bf2-45ba216df6fb" containerName="main"
Nov 03 08:29:23.945084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944833    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="e724a396-646c-42bb-9b59-d7ff618e07f8" containerName="istio-proxy"
Nov 03 08:29:23.945084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944844    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fcd6530-2913-45a2-a36b-697f56d42909" containerName="payload-pre-processing"
Nov 03 08:29:23.945084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944854    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="56258879-0d2c-457c-a0d2-0e7c378bed50" containerName="payload-pre-processing"
Nov 03 08:29:23.945084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944863    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0822706-f191-47c7-8c87-e373872590e3" containerName="istio-proxy"
Nov 03 08:29:23.945084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944873    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="216f610f-a3c5-4967-949d-fbabca17ed1a" containerName="payload-processing"
Nov 03 08:29:23.945084 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.944882    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="55536e9b-0a23-40c8-b9d5-0e9fe2a065c0" containerName="revoke-keys"
Nov 03 08:29:23.948586 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.948567    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:23.955308 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:23.955285    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"]
Nov 03 08:29:24.046902 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.046873    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a86a6470-9786-4e2d-8668-62ce0dbbb596-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.047015 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.046940    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a86a6470-9786-4e2d-8668-62ce0dbbb596-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.047121 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.047094    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a86a6470-9786-4e2d-8668-62ce0dbbb596-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.047170 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.047140    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a86a6470-9786-4e2d-8668-62ce0dbbb596-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.047205 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.047173    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a86a6470-9786-4e2d-8668-62ce0dbbb596-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.047273 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.047257    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjjgg\" (UniqueName: \"kubernetes.io/projected/a86a6470-9786-4e2d-8668-62ce0dbbb596-kube-api-access-fjjgg\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.047332 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.047310    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a86a6470-9786-4e2d-8668-62ce0dbbb596-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.047406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.047392    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a86a6470-9786-4e2d-8668-62ce0dbbb596-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.047502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.047489    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a86a6470-9786-4e2d-8668-62ce0dbbb596-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.147995 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.147973    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a86a6470-9786-4e2d-8668-62ce0dbbb596-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.148207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.148017    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjjgg\" (UniqueName: \"kubernetes.io/projected/a86a6470-9786-4e2d-8668-62ce0dbbb596-kube-api-access-fjjgg\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.148207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.148046    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a86a6470-9786-4e2d-8668-62ce0dbbb596-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.148207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.148066    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a86a6470-9786-4e2d-8668-62ce0dbbb596-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.148207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.148092    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a86a6470-9786-4e2d-8668-62ce0dbbb596-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.148207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.148121    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a86a6470-9786-4e2d-8668-62ce0dbbb596-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.148207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.148174    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a86a6470-9786-4e2d-8668-62ce0dbbb596-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.148207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.148207    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a86a6470-9786-4e2d-8668-62ce0dbbb596-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.148612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.148235    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a86a6470-9786-4e2d-8668-62ce0dbbb596-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.148612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.148497    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a86a6470-9786-4e2d-8668-62ce0dbbb596-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.148612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.148575    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a86a6470-9786-4e2d-8668-62ce0dbbb596-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.148612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.148603    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a86a6470-9786-4e2d-8668-62ce0dbbb596-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.148780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.148683    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a86a6470-9786-4e2d-8668-62ce0dbbb596-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.148839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.148783    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a86a6470-9786-4e2d-8668-62ce0dbbb596-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.150380 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.150360    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a86a6470-9786-4e2d-8668-62ce0dbbb596-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.150718 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.150697    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a86a6470-9786-4e2d-8668-62ce0dbbb596-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.154906 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.154880    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjjgg\" (UniqueName: \"kubernetes.io/projected/a86a6470-9786-4e2d-8668-62ce0dbbb596-kube-api-access-fjjgg\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.154906 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.154891    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a86a6470-9786-4e2d-8668-62ce0dbbb596-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-8rcwm\" (UID: \"a86a6470-9786-4e2d-8668-62ce0dbbb596\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.267406 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.267353    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:24.547826 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.547802    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"]
Nov 03 08:29:24.549942 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:29:24.549907    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda86a6470_9786_4e2d_8668_62ce0dbbb596.slice/crio-a5fd6512e37102841867b88c99d9db0985c542765fbbbf7f93c33009941fc1e8 WatchSource:0}: Error finding container a5fd6512e37102841867b88c99d9db0985c542765fbbbf7f93c33009941fc1e8: Status 404 returned error can't find the container with id a5fd6512e37102841867b88c99d9db0985c542765fbbbf7f93c33009941fc1e8
Nov 03 08:29:24.556901 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.556870    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:29:24.556994 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.556934    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:29:24.556994 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:24.556966    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:29:25.281836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:25.281671    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm" event={"ID":"a86a6470-9786-4e2d-8668-62ce0dbbb596","Type":"ContainerStarted","Data":"cc1f432092bb89cd785abebcbbae94f35a740204574b6dc17e0c6dbbbed47d80"}
Nov 03 08:29:25.281836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:25.281748    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm" event={"ID":"a86a6470-9786-4e2d-8668-62ce0dbbb596","Type":"ContainerStarted","Data":"a5fd6512e37102841867b88c99d9db0985c542765fbbbf7f93c33009941fc1e8"}
Nov 03 08:29:25.304086 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:25.304038    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm" podStartSLOduration=2.304024352 podStartE2EDuration="2.304024352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:29:25.300592456 +0000 UTC m=+2163.415056387" watchObservedRunningTime="2026-11-03 08:29:25.304024352 +0000 UTC m=+2163.418488282"
Nov 03 08:29:26.279779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:26.274726    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:26.279779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:26.274907    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8rcwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Nov 03 08:29:26.279779 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:26.274955    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm" podUID="a86a6470-9786-4e2d-8668-62ce0dbbb596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Nov 03 08:29:27.277188 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:27.277148    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8rcwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Nov 03 08:29:27.277823 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:27.277789    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm" podUID="a86a6470-9786-4e2d-8668-62ce0dbbb596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Nov 03 08:29:28.278780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:28.275848    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8rcwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Nov 03 08:29:28.278780 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:28.275912    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm" podUID="a86a6470-9786-4e2d-8668-62ce0dbbb596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Nov 03 08:29:29.277089 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:29.277055    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8rcwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Nov 03 08:29:29.277391 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:29.277361    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm" podUID="a86a6470-9786-4e2d-8668-62ce0dbbb596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Nov 03 08:29:30.273782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:30.273067    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8rcwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Nov 03 08:29:30.273782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:30.273124    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm" podUID="a86a6470-9786-4e2d-8668-62ce0dbbb596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Nov 03 08:29:31.268280 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:31.268243    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8rcwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body=
Nov 03 08:29:31.268460 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:31.268300    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm" podUID="a86a6470-9786-4e2d-8668-62ce0dbbb596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused"
Nov 03 08:29:33.269439 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:33.269399    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8rcwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:29:33.270000 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:33.269458    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm" podUID="a86a6470-9786-4e2d-8668-62ce0dbbb596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:29:34.269719 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:34.269672    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-8rcwm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:29:34.270255 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:34.269729    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm" podUID="a86a6470-9786-4e2d-8668-62ce0dbbb596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:29:34.770739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:34.770702    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:34.771254 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:34.771232    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:34.771908 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:34.771893    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-8rcwm"
Nov 03 08:29:42.025415 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:42.025382    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5"]
Nov 03 08:29:42.030836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:42.030808    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8"]
Nov 03 08:29:42.031080 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:42.031061    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a06b773-drqz5"]
Nov 03 08:29:42.033133 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:42.033113    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a14a1e4-jpxs8"]
Nov 03 08:29:44.024185 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:44.024150    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c070983-a6e2-41ce-aff5-8526f7d99375" path="/var/lib/kubelet/pods/0c070983-a6e2-41ce-aff5-8526f7d99375/volumes"
Nov 03 08:29:44.024568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:44.024543    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0820001-c7d2-49f8-afa6-79038e19c795" path="/var/lib/kubelet/pods/d0820001-c7d2-49f8-afa6-79038e19c795/volumes"
Nov 03 08:29:52.053132 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:29:52.053094    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:29:59.023327 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:59.023287    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp"]
Nov 03 08:29:59.027354 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:59.027330    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm"]
Nov 03 08:29:59.029633 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:59.029598    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-157f42c1-mc5rp"]
Nov 03 08:29:59.031575 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:29:59.031556    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9162aaae-f52hm"]
Nov 03 08:30:00.024633 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:30:00.024596    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42228923-0917-4753-af54-c3fc849be091" path="/var/lib/kubelet/pods/42228923-0917-4753-af54-c3fc849be091/volumes"
Nov 03 08:30:00.025100 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:30:00.025083    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="506fb467-77cf-4b49-8e99-b787b98b490f" path="/var/lib/kubelet/pods/506fb467-77cf-4b49-8e99-b787b98b490f/volumes"
Nov 03 08:30:14.830648 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:30:14.830560    5070 scope.go:122] "RemoveContainer" containerID="8456e11dd25c095191c6ad4110b287f538a7ebac3e76d00be6ea9786a7c3f70e"
Nov 03 08:30:14.841226 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:30:14.841192    5070 scope.go:122] "RemoveContainer" containerID="d481e7e7b611192ca40c18e95f5272f592b515079aa1f8bb09e9a6ab6ad9def3"
Nov 03 08:30:14.849859 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:30:14.849840    5070 scope.go:122] "RemoveContainer" containerID="82255fdd3cdedf7f896fc6322dfe16b6b3df9fd8e66876be1564e2cd769ac487"
Nov 03 08:30:14.859030 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:30:14.859009    5070 scope.go:122] "RemoveContainer" containerID="86bacb13204c4dd5b1b3ac4fd8ef7496baae92694558254546ecffe85da53bb9"
Nov 03 08:30:14.868115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:30:14.868085    5070 scope.go:122] "RemoveContainer" containerID="d3021274e97b01f31e9a7d9d62be1b1b7649c569433a8fdbed24423511cf841a"
Nov 03 08:30:14.876614 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:30:14.876594    5070 scope.go:122] "RemoveContainer" containerID="8ea162f343930666669df4c5c4596c179dc694807f3e78b1bbf4fcdd7911ca04"
Nov 03 08:30:14.884620 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:30:14.884602    5070 scope.go:122] "RemoveContainer" containerID="26f562bf40571169c9c466d011e4e469e67f0b9f73579faf7a7c3c764e05e69d"
Nov 03 08:30:14.892751 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:30:14.892732    5070 scope.go:122] "RemoveContainer" containerID="270f5e4064f4a48f7a1821756bdc6d0c984e03c9f80543bb13a67148f9b2970a"
Nov 03 08:30:14.901170 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:30:14.901151    5070 scope.go:122] "RemoveContainer" containerID="bf121ce95868d2df9d674aa325b9bd9b9f6f03832b90dd857260a3150136a6be"
Nov 03 08:30:14.909337 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:30:14.909321    5070 scope.go:122] "RemoveContainer" containerID="4a1999b3bfc18a679661b7a31750322a8efd2dcb638b078ee570a69b745ea8da"
Nov 03 08:30:14.917388 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:30:14.917373    5070 scope.go:122] "RemoveContainer" containerID="5e6ca19ee13217660a024a0597350c1d305d864db6499eea820295dd5614da9e"
Nov 03 08:30:15.587515 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:30:15.587481    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:30:22.056698 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:30:22.056660    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:30:33.815923 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:30:33.815892    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:30:38.205784 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:30:38.204454    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:30:52.050395 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:30:52.050360    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:30:52.457770 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:30:52.457725    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:31:05.088040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:05.088007    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:31:14.987859 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:14.987827    5070 scope.go:122] "RemoveContainer" containerID="2d548fe6af73f0948b4b2e1491c2aa364fe93cf53a5ed11efbd293807aad58bc"
Nov 03 08:31:14.997998 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:14.997980    5070 scope.go:122] "RemoveContainer" containerID="4fc053c261ba3813371ae3b9132f359d0ba3c6133718d9eac2ecd7cae39663b8"
Nov 03 08:31:15.010055 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:15.010023    5070 scope.go:122] "RemoveContainer" containerID="2397148c59a145e68d69f487d17e256858d0f22cff879614978964f940aa4219"
Nov 03 08:31:15.018249 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:15.018230    5070 scope.go:122] "RemoveContainer" containerID="0d99ce8d05353067d19e91966595d6585d95d9bf9244328113c135e5a14b83c6"
Nov 03 08:31:15.026333 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:15.026316    5070 scope.go:122] "RemoveContainer" containerID="2da2634436e9ba0c3aa4851cc650c7c380d558837d1e07443abb0127d008625f"
Nov 03 08:31:15.035822 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:15.035804    5070 scope.go:122] "RemoveContainer" containerID="7e5b490de34ee80bd519cb3455730a44cf45e883f30f480cc6bf0e599e7ba9e6"
Nov 03 08:31:22.050482 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:31:22.050447    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:31:24.025273 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:24.025241    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb3a58d0-ql8t6"]
Nov 03 08:31:24.026716 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:24.026693    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb3a58d0-ql8t6"]
Nov 03 08:31:26.026156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:26.026111    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3868d631-022d-4a08-b8aa-1d70aa1ebc01" path="/var/lib/kubelet/pods/3868d631-022d-4a08-b8aa-1d70aa1ebc01/volumes"
Nov 03 08:31:29.510354 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:29.510321    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:31:35.024782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:35.024671    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh"]
Nov 03 08:31:35.027028 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:35.026997    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-025362e9-4ghsh"]
Nov 03 08:31:36.024418 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:36.024387    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e2bb450-f2a0-41e2-90bb-9600e443f7b2" path="/var/lib/kubelet/pods/5e2bb450-f2a0-41e2-90bb-9600e443f7b2/volumes"
Nov 03 08:31:38.026872 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:38.026835    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds"]
Nov 03 08:31:38.029120 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:38.029097    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36111c31-2j4ds"]
Nov 03 08:31:39.689948 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:39.689919    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:31:40.034152 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:40.034077    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18d88475-e8f2-42dc-b2c9-420e69c64636" path="/var/lib/kubelet/pods/18d88475-e8f2-42dc-b2c9-420e69c64636/volumes"
Nov 03 08:31:44.834920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:44.834881    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:31:52.024942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:52.024905    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg"]
Nov 03 08:31:52.024942 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:52.024946    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b02faba-j8hkg"]
Nov 03 08:31:52.054120 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:31:52.054075    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:31:54.024267 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:31:54.024232    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="510e7368-f5d8-4c33-b226-5c996f9b81c0" path="/var/lib/kubelet/pods/510e7368-f5d8-4c33-b226-5c996f9b81c0/volumes"
Nov 03 08:32:00.078007 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:32:00.077977    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:32:04.515781 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:32:04.515197    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:32:18.166824 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:32:18.166790    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:32:22.129684 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:32:22.129646    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:32:23.727352 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:32:23.727316    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:32:30.024818 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:32:30.024788    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5"]
Nov 03 08:32:30.024818 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:32:30.024826    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc1552c1-6jwk5"]
Nov 03 08:32:32.025204 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:32:32.025162    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e940da6d-ee8d-42f1-bdf2-e580a3705fab" path="/var/lib/kubelet/pods/e940da6d-ee8d-42f1-bdf2-e580a3705fab/volumes"
Nov 03 08:32:36.684118 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:32:36.684066    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:32:40.032794 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:32:40.032747    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ccb783-pg7wz"]
Nov 03 08:32:40.033191 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:32:40.032826    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ccb783-pg7wz"]
Nov 03 08:32:40.878782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:32:40.877685    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:32:42.023920 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:32:42.023888    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1fa3109-88e6-427c-8239-f1f1604231f9" path="/var/lib/kubelet/pods/c1fa3109-88e6-427c-8239-f1f1604231f9/volumes"
Nov 03 08:32:49.020606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:32:49.020574    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b929697f-6btsc"]
Nov 03 08:32:49.023225 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:32:49.023194    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b929697f-6btsc"]
Nov 03 08:32:50.033354 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:32:50.033319    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e569b99d-12a2-455b-9f87-45e229768318" path="/var/lib/kubelet/pods/e569b99d-12a2-455b-9f87-45e229768318/volumes"
Nov 03 08:32:50.033741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:32:50.033601    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5"]
Nov 03 08:32:50.033741 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:32:50.033624    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a1c6b34-ggnf5"]
Nov 03 08:32:52.024867 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:32:52.024830    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7eb5a76-5765-468d-972b-08ce645c4fa8" path="/var/lib/kubelet/pods/c7eb5a76-5765-468d-972b-08ce645c4fa8/volumes"
Nov 03 08:32:52.052521 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:32:52.052485    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:32:54.371366 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:32:54.371332    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:33:04.943601 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:04.943516    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:33:10.132789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:10.132741    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:33:15.099466 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:15.099429    5070 scope.go:122] "RemoveContainer" containerID="a83b915f0fc98c2060e6d64c080f44c9ab1ade1e1a510c4943b1870ed455af90"
Nov 03 08:33:15.110274 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:15.110244    5070 scope.go:122] "RemoveContainer" containerID="0619387f1aeafe6242c20e572bb78364964948700157b66b8585af08a38b8723"
Nov 03 08:33:15.128431 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:15.128403    5070 scope.go:122] "RemoveContainer" containerID="6ef495fa6ceddb56d57d1480cf4661fdc7b8a24205c27e72324384f8b8f8ca9f"
Nov 03 08:33:15.139407 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:15.139383    5070 scope.go:122] "RemoveContainer" containerID="3c2016c13fd48186b9af89497995b7770d5789016ee1cb92f5a22ce6e80f06df"
Nov 03 08:33:15.150073 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:15.150049    5070 scope.go:122] "RemoveContainer" containerID="d18f506bb2de0a8e812b8e168c256bf8aadb09fdca38cb22b50056dfdd62dd27"
Nov 03 08:33:15.160422 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:15.160400    5070 scope.go:122] "RemoveContainer" containerID="a2cafe5cdc571bcb38765a29cb262319d935b9223c6d13313952f7dbb8efac3a"
Nov 03 08:33:15.168988 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:15.168969    5070 scope.go:122] "RemoveContainer" containerID="37143d286f2ad40097d44c27f85e43ad89e54af3d7b94137a6662c74d825882a"
Nov 03 08:33:22.050956 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:33:22.050916    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:33:22.650690 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:22.650662    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/3.log"
Nov 03 08:33:22.652421 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:22.652396    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/3.log"
Nov 03 08:33:22.659931 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:22.659910    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-n28db_5738916f-8202-40ed-823e-90f7c9cc4ec9/csi-driver/0.log"
Nov 03 08:33:22.661671 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:22.661649    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-n28db_5738916f-8202-40ed-823e-90f7c9cc4ec9/csi-driver/0.log"
Nov 03 08:33:26.491724 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:26.491687    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:33:30.303014 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:30.302977    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:33:33.015414 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.015384    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"]
Nov 03 08:33:33.015884 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.015578    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z" podUID="83c0eb74-3811-4db9-b783-b1b9a14ee7b5" containerName="discovery" containerID="cri-o://f25ba54996fdbc7c20ebec38667dc32d5f81de3f783abeffffd12150c17e991d" gracePeriod=30
Nov 03 08:33:33.172003 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.171967    5070 generic.go:356] "Generic (PLEG): container finished" podID="83c0eb74-3811-4db9-b783-b1b9a14ee7b5" containerID="f25ba54996fdbc7c20ebec38667dc32d5f81de3f783abeffffd12150c17e991d" exitCode=0
Nov 03 08:33:33.172190 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.172034    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z" event={"ID":"83c0eb74-3811-4db9-b783-b1b9a14ee7b5","Type":"ContainerDied","Data":"f25ba54996fdbc7c20ebec38667dc32d5f81de3f783abeffffd12150c17e991d"}
Nov 03 08:33:33.285204 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.285179    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:33:33.407480 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.407451    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-kube-api-access-bvcnm\" (UniqueName: \"kubernetes.io/projected/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-kube-api-access-bvcnm\") pod \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") "
Nov 03 08:33:33.407628 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.407492    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-local-certs\") pod \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") "
Nov 03 08:33:33.407628 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.407536    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-kubeconfig\") pod \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") "
Nov 03 08:33:33.407628 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.407558    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-csr-ca-configmap\") pod \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") "
Nov 03 08:33:33.407628 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.407573    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-cacerts\" (UniqueName: \"kubernetes.io/secret/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-cacerts\") pod \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") "
Nov 03 08:33:33.407889 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.407629    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-csr-dns-cert\") pod \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") "
Nov 03 08:33:33.408101 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.408074    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-csr-ca-configmap" pod "83c0eb74-3811-4db9-b783-b1b9a14ee7b5" (UID: "83c0eb74-3811-4db9-b783-b1b9a14ee7b5"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:33:33.408276 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.408111    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-token\" (UniqueName: \"kubernetes.io/projected/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-token\") pod \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\" (UID: \"83c0eb74-3811-4db9-b783-b1b9a14ee7b5\") "
Nov 03 08:33:33.408695 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.408660    5070 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-csr-ca-configmap\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:33:33.410003 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.409976    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-kube-api-access-bvcnm" pod "83c0eb74-3811-4db9-b783-b1b9a14ee7b5" (UID: "83c0eb74-3811-4db9-b783-b1b9a14ee7b5"). InnerVolumeSpecName "kube-api-access-bvcnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:33:33.410130 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.410106    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-token" pod "83c0eb74-3811-4db9-b783-b1b9a14ee7b5" (UID: "83c0eb74-3811-4db9-b783-b1b9a14ee7b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:33:33.410194 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.410163    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-local-certs" pod "83c0eb74-3811-4db9-b783-b1b9a14ee7b5" (UID: "83c0eb74-3811-4db9-b783-b1b9a14ee7b5"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:33:33.410267 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.410242    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-kubeconfig" pod "83c0eb74-3811-4db9-b783-b1b9a14ee7b5" (UID: "83c0eb74-3811-4db9-b783-b1b9a14ee7b5"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:33:33.410333 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.410284    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-csr-dns-cert" pod "83c0eb74-3811-4db9-b783-b1b9a14ee7b5" (UID: "83c0eb74-3811-4db9-b783-b1b9a14ee7b5"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:33:33.410388 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.410351    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-cacerts" pod "83c0eb74-3811-4db9-b783-b1b9a14ee7b5" (UID: "83c0eb74-3811-4db9-b783-b1b9a14ee7b5"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:33:33.509412 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.509389    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvcnm\" (UniqueName: \"kubernetes.io/projected/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-kube-api-access-bvcnm\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:33:33.509412 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.509411    5070 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-local-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:33:33.509543 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.509423    5070 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-kubeconfig\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:33:33.509543 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.509432    5070 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-cacerts\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:33:33.509543 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.509441    5070 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-csr-dns-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:33:33.509543 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:33.509449    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83c0eb74-3811-4db9-b783-b1b9a14ee7b5-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:33:34.181218 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:34.181175    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z" event={"ID":"83c0eb74-3811-4db9-b783-b1b9a14ee7b5","Type":"ContainerDied","Data":"7dd0d2193469fe9eabeba8d93489ae89a1204177225064ef18cabfeeac48939d"}
Nov 03 08:33:34.181680 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:34.181236    5070 scope.go:122] "RemoveContainer" containerID="f25ba54996fdbc7c20ebec38667dc32d5f81de3f783abeffffd12150c17e991d"
Nov 03 08:33:34.181680 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:34.181362    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"
Nov 03 08:33:34.197923 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:34.197899    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"]
Nov 03 08:33:34.201515 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:34.201492    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-26r8z"]
Nov 03 08:33:35.691023 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:35.690991    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:33:36.024615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:36.024544    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83c0eb74-3811-4db9-b783-b1b9a14ee7b5" path="/var/lib/kubelet/pods/83c0eb74-3811-4db9-b783-b1b9a14ee7b5/volumes"
Nov 03 08:33:40.936474 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:40.936440    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:33:43.026365 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:43.026325    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr"]
Nov 03 08:33:43.028309 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:43.028288    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6249b7ca-zlvsr"]
Nov 03 08:33:44.024080 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:44.024044    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff7a503-9cf4-4538-8bd5-da7ad324a6af" path="/var/lib/kubelet/pods/aff7a503-9cf4-4538-8bd5-da7ad324a6af/volumes"
Nov 03 08:33:47.022261 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:47.022207    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm"]
Nov 03 08:33:47.027750 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:47.027719    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfd9bcb3-lgxxm"]
Nov 03 08:33:48.023775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:48.023718    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55536e9b-0a23-40c8-b9d5-0e9fe2a065c0" path="/var/lib/kubelet/pods/55536e9b-0a23-40c8-b9d5-0e9fe2a065c0/volumes"
Nov 03 08:33:52.051853 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:33:52.051819    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:33:57.458910 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:33:57.458878    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:34:07.694695 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:34:07.694662    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:34:15.217280 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:34:15.217249    5070 scope.go:122] "RemoveContainer" containerID="f7234f2291356a29b53722e4fddf2edb419fb84c5344deb364587403a800ad31"
Nov 03 08:34:15.227501 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:34:15.227483    5070 scope.go:122] "RemoveContainer" containerID="6a513ac968a98bf31250dd1c015f173cec39a0989123bb402b89a08d7bdd1f48"
Nov 03 08:34:15.235472 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:34:15.235457    5070 scope.go:122] "RemoveContainer" containerID="f75911299efc04c7bdd5965f05f0fb5164e6d6c12246978597949704547c02b7"
Nov 03 08:34:15.243582 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:34:15.243559    5070 scope.go:122] "RemoveContainer" containerID="8c54ba82756debfd77e137822265b74fcb5bd18569130a897749e4d1789481b4"
Nov 03 08:34:15.251621 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:34:15.251603    5070 scope.go:122] "RemoveContainer" containerID="d6f2c6e93cc28712da8e521fba2ef4cb373740f44f26a4cb9320f6cbf7556ce7"
Nov 03 08:34:15.259606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:34:15.259587    5070 scope.go:122] "RemoveContainer" containerID="3b0319880bd4d244a1bf16da1c2705ea21cc4bd41538bc149bc90aa456a68a5a"
Nov 03 08:34:15.267641 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:34:15.267621    5070 scope.go:122] "RemoveContainer" containerID="0999cd513af3becb230dfc5833512911040d67852d69a7e4fba157333c7ab733"
Nov 03 08:34:15.275193 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:34:15.275177    5070 scope.go:122] "RemoveContainer" containerID="9eac18a25fa32a547926a446e24a0c0abd72a19e09b37490d8576f60dad28b21"
Nov 03 08:34:15.283179 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:34:15.283164    5070 scope.go:122] "RemoveContainer" containerID="418c79eb23f708b0c8c3a86433b336a2384f4059602fd85d31d0e596009ae455"
Nov 03 08:34:15.290702 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:34:15.290686    5070 scope.go:122] "RemoveContainer" containerID="12590141bcff365cdfdf6ed9de32d4570b73fe7b8c8c086750413a608a1d482d"
Nov 03 08:34:22.050052 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:34:22.050018    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:34:23.395849 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:34:23.395818    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:34:28.070502 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:34:28.070468    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:34:40.183872 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:34:40.183788    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:34:52.050517 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:34:52.050479    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:34:56.193396 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:34:56.193356    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:35:12.697036 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:12.696995    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:35:15.363034 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:15.363002    5070 scope.go:122] "RemoveContainer" containerID="297f7d3cb65aac8da005abcf5293af6d9961600208519db880c01deafeb0c5f7"
Nov 03 08:35:15.372188 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:15.372166    5070 scope.go:122] "RemoveContainer" containerID="0f18ccbd7c5bd67a07e2992fa79b1424b54ababd8a1d620cd68b6d1510541ade"
Nov 03 08:35:15.380165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:15.380147    5070 scope.go:122] "RemoveContainer" containerID="12acdd43fcff37bcc1116f28bc0eb56ec458570142a6c8ad8b837b9def88387f"
Nov 03 08:35:15.387836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:15.387814    5070 scope.go:122] "RemoveContainer" containerID="9090debcd715791190c07d45effaa2b3fb6c4e966f1eb2ee32caf30b67a4225a"
Nov 03 08:35:22.055711 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:35:22.055676    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:35:25.835567 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:25.835533    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc"]
Nov 03 08:35:25.835983 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:25.835964    5070 cpu_manager.go:414] "containerMap: removing container" podUID="83c0eb74-3811-4db9-b783-b1b9a14ee7b5" containerName="discovery"
Nov 03 08:35:25.835983 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:25.835976    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83c0eb74-3811-4db9-b783-b1b9a14ee7b5" containerName="discovery"
Nov 03 08:35:25.836080 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:25.836053    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="83c0eb74-3811-4db9-b783-b1b9a14ee7b5" containerName="discovery"
Nov 03 08:35:25.839162 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:25.839141    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc"
Nov 03 08:35:25.844584 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:25.844559    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:35:25.844826 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:25.844633    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:35:25.846852 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:25.846827    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc"]
Nov 03 08:35:25.855081 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:25.855047    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:35:25.855192 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:25.855153    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:35:25.897223 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:25.897193    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4412ebe-7jrfc\" (UID: \"b1a5d028-62e6-41ce-ab11-aa12f3cdb63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc"
Nov 03 08:35:25.897334 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:25.897302    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4412ebe-7jrfc\" (UID: \"b1a5d028-62e6-41ce-ab11-aa12f3cdb63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc"
Nov 03 08:35:25.897403 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:25.897376    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bn2l\" (UniqueName: \"kubernetes.io/projected/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a-kube-api-access-4bn2l\") pod \"maas-api-revoke-sub-c4412ebe-7jrfc\" (UID: \"b1a5d028-62e6-41ce-ab11-aa12f3cdb63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc"
Nov 03 08:35:25.998639 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:25.998602    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bn2l\" (UniqueName: \"kubernetes.io/projected/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a-kube-api-access-4bn2l\") pod \"maas-api-revoke-sub-c4412ebe-7jrfc\" (UID: \"b1a5d028-62e6-41ce-ab11-aa12f3cdb63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc"
Nov 03 08:35:25.998748 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:25.998662    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4412ebe-7jrfc\" (UID: \"b1a5d028-62e6-41ce-ab11-aa12f3cdb63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc"
Nov 03 08:35:25.998748 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:25.998714    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4412ebe-7jrfc\" (UID: \"b1a5d028-62e6-41ce-ab11-aa12f3cdb63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc"
Nov 03 08:35:25.999278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:25.999257    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4412ebe-7jrfc\" (UID: \"b1a5d028-62e6-41ce-ab11-aa12f3cdb63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc"
Nov 03 08:35:26.001017 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:26.000996    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4412ebe-7jrfc\" (UID: \"b1a5d028-62e6-41ce-ab11-aa12f3cdb63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc"
Nov 03 08:35:26.006112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:26.006094    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bn2l\" (UniqueName: \"kubernetes.io/projected/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a-kube-api-access-4bn2l\") pod \"maas-api-revoke-sub-c4412ebe-7jrfc\" (UID: \"b1a5d028-62e6-41ce-ab11-aa12f3cdb63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc"
Nov 03 08:35:26.153540 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:26.153517    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc"
Nov 03 08:35:26.437774 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:26.437680    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc"]
Nov 03 08:35:26.441754 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:35:26.441725    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a5d028_62e6_41ce_ab11_aa12f3cdb63a.slice/crio-24ab9d7149c8e8b68d7446b62207f3b8aa50f5e9fd50397b7b9613b684bb03b1 WatchSource:0}: Error finding container 24ab9d7149c8e8b68d7446b62207f3b8aa50f5e9fd50397b7b9613b684bb03b1: Status 404 returned error can't find the container with id 24ab9d7149c8e8b68d7446b62207f3b8aa50f5e9fd50397b7b9613b684bb03b1
Nov 03 08:35:26.879944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:26.879917    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc" event={"ID":"b1a5d028-62e6-41ce-ab11-aa12f3cdb63a","Type":"ContainerStarted","Data":"f34578bf493c1033cab4b9ba1388acc1ae280717c38c77bbca6fc4a22411126c"}
Nov 03 08:35:26.880303 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:26.879967    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc" event={"ID":"b1a5d028-62e6-41ce-ab11-aa12f3cdb63a","Type":"ContainerStarted","Data":"24ab9d7149c8e8b68d7446b62207f3b8aa50f5e9fd50397b7b9613b684bb03b1"}
Nov 03 08:35:26.896131 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:26.896091    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc" podStartSLOduration=1.8960786490000001 podStartE2EDuration="1.896078649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:35:26.893932404 +0000 UTC m=+2525.008396348" watchObservedRunningTime="2026-11-03 08:35:26.896078649 +0000 UTC m=+2525.010542579"
Nov 03 08:35:27.887555 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:27.887522    5070 generic.go:356] "Generic (PLEG): container finished" podID="b1a5d028-62e6-41ce-ab11-aa12f3cdb63a" containerID="f34578bf493c1033cab4b9ba1388acc1ae280717c38c77bbca6fc4a22411126c" exitCode=0
Nov 03 08:35:27.887987 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:27.887605    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc" event={"ID":"b1a5d028-62e6-41ce-ab11-aa12f3cdb63a","Type":"ContainerDied","Data":"f34578bf493c1033cab4b9ba1388acc1ae280717c38c77bbca6fc4a22411126c"}
Nov 03 08:35:29.019229 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:29.019208    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc"
Nov 03 08:35:29.125807 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:29.125778    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a-maas-api-openshift-ca\") pod \"b1a5d028-62e6-41ce-ab11-aa12f3cdb63a\" (UID: \"b1a5d028-62e6-41ce-ab11-aa12f3cdb63a\") "
Nov 03 08:35:29.125965 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:29.125848    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a-maas-api-xks-ca\") pod \"b1a5d028-62e6-41ce-ab11-aa12f3cdb63a\" (UID: \"b1a5d028-62e6-41ce-ab11-aa12f3cdb63a\") "
Nov 03 08:35:29.126025 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:29.126005    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a-kube-api-access-4bn2l\" (UniqueName: \"kubernetes.io/projected/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a-kube-api-access-4bn2l\") pod \"b1a5d028-62e6-41ce-ab11-aa12f3cdb63a\" (UID: \"b1a5d028-62e6-41ce-ab11-aa12f3cdb63a\") "
Nov 03 08:35:29.126161 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:29.126141    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a-maas-api-openshift-ca" pod "b1a5d028-62e6-41ce-ab11-aa12f3cdb63a" (UID: "b1a5d028-62e6-41ce-ab11-aa12f3cdb63a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:35:29.126340 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:29.126316    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:29.127971 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:29.127949    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a-maas-api-xks-ca" pod "b1a5d028-62e6-41ce-ab11-aa12f3cdb63a" (UID: "b1a5d028-62e6-41ce-ab11-aa12f3cdb63a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:35:29.128733 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:29.128700    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a-kube-api-access-4bn2l" pod "b1a5d028-62e6-41ce-ab11-aa12f3cdb63a" (UID: "b1a5d028-62e6-41ce-ab11-aa12f3cdb63a"). InnerVolumeSpecName "kube-api-access-4bn2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:35:29.227341 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:29.227261    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bn2l\" (UniqueName: \"kubernetes.io/projected/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a-kube-api-access-4bn2l\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:29.227341 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:29.227291    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:29.899573 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:29.899541    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc"
Nov 03 08:35:29.899573 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:29.899567    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc" event={"ID":"b1a5d028-62e6-41ce-ab11-aa12f3cdb63a","Type":"ContainerDied","Data":"24ab9d7149c8e8b68d7446b62207f3b8aa50f5e9fd50397b7b9613b684bb03b1"}
Nov 03 08:35:29.899961 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:29.899615    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24ab9d7149c8e8b68d7446b62207f3b8aa50f5e9fd50397b7b9613b684bb03b1"
Nov 03 08:35:36.891215 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:36.891179    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs"]
Nov 03 08:35:36.891649 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:36.891635    5070 cpu_manager.go:414] "containerMap: removing container" podUID="b1a5d028-62e6-41ce-ab11-aa12f3cdb63a" containerName="revoke-keys"
Nov 03 08:35:36.891696 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:36.891651    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1a5d028-62e6-41ce-ab11-aa12f3cdb63a" containerName="revoke-keys"
Nov 03 08:35:36.891737 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:36.891722    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1a5d028-62e6-41ce-ab11-aa12f3cdb63a" containerName="revoke-keys"
Nov 03 08:35:36.895040 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:36.895025    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs"
Nov 03 08:35:36.900384 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:36.900361    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:35:36.900783 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:36.900732    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:35:36.900948 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:36.900751    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:35:36.901052 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:36.900782    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:35:36.903709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:36.903685    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs"]
Nov 03 08:35:36.995131 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:36.995106    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda9066d-68ff-4e5d-9272-ef793d05228b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs\" (UID: \"dda9066d-68ff-4e5d-9272-ef793d05228b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs"
Nov 03 08:35:36.995282 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:36.995155    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda9066d-68ff-4e5d-9272-ef793d05228b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs\" (UID: \"dda9066d-68ff-4e5d-9272-ef793d05228b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs"
Nov 03 08:35:36.995366 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:36.995317    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cl6g\" (UniqueName: \"kubernetes.io/projected/dda9066d-68ff-4e5d-9272-ef793d05228b-kube-api-access-2cl6g\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs\" (UID: \"dda9066d-68ff-4e5d-9272-ef793d05228b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs"
Nov 03 08:35:37.096046 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:37.096016    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda9066d-68ff-4e5d-9272-ef793d05228b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs\" (UID: \"dda9066d-68ff-4e5d-9272-ef793d05228b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs"
Nov 03 08:35:37.096191 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:37.096166    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cl6g\" (UniqueName: \"kubernetes.io/projected/dda9066d-68ff-4e5d-9272-ef793d05228b-kube-api-access-2cl6g\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs\" (UID: \"dda9066d-68ff-4e5d-9272-ef793d05228b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs"
Nov 03 08:35:37.096260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:37.096209    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda9066d-68ff-4e5d-9272-ef793d05228b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs\" (UID: \"dda9066d-68ff-4e5d-9272-ef793d05228b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs"
Nov 03 08:35:37.096704 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:37.096678    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda9066d-68ff-4e5d-9272-ef793d05228b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs\" (UID: \"dda9066d-68ff-4e5d-9272-ef793d05228b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs"
Nov 03 08:35:37.098492 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:37.098468    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda9066d-68ff-4e5d-9272-ef793d05228b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs\" (UID: \"dda9066d-68ff-4e5d-9272-ef793d05228b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs"
Nov 03 08:35:37.103092 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:37.103072    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cl6g\" (UniqueName: \"kubernetes.io/projected/dda9066d-68ff-4e5d-9272-ef793d05228b-kube-api-access-2cl6g\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs\" (UID: \"dda9066d-68ff-4e5d-9272-ef793d05228b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs"
Nov 03 08:35:37.209951 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:37.209902    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs"
Nov 03 08:35:37.482890 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:37.482861    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs"]
Nov 03 08:35:37.485025 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:35:37.484991    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda9066d_68ff_4e5d_9272_ef793d05228b.slice/crio-4d929d9d5a718afe8757422ee4134495ddef6b5af1624ad8813f75353ffaa3e3 WatchSource:0}: Error finding container 4d929d9d5a718afe8757422ee4134495ddef6b5af1624ad8813f75353ffaa3e3: Status 404 returned error can't find the container with id 4d929d9d5a718afe8757422ee4134495ddef6b5af1624ad8813f75353ffaa3e3
Nov 03 08:35:37.758668 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:37.758643    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:35:37.942829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:37.942787    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs" event={"ID":"dda9066d-68ff-4e5d-9272-ef793d05228b","Type":"ContainerStarted","Data":"34bf3e951da2b449323e4434eb242d11ac21488c559e65088705a482cc7b0913"}
Nov 03 08:35:37.942829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:37.942833    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs" event={"ID":"dda9066d-68ff-4e5d-9272-ef793d05228b","Type":"ContainerStarted","Data":"4d929d9d5a718afe8757422ee4134495ddef6b5af1624ad8813f75353ffaa3e3"}
Nov 03 08:35:37.957416 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:37.957361    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs" podStartSLOduration=1.957345213 podStartE2EDuration="1.957345213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:35:37.956835341 +0000 UTC m=+2536.071299264" watchObservedRunningTime="2026-11-03 08:35:37.957345213 +0000 UTC m=+2536.071809149"
Nov 03 08:35:38.950944 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:38.950893    5070 generic.go:356] "Generic (PLEG): container finished" podID="dda9066d-68ff-4e5d-9272-ef793d05228b" containerID="34bf3e951da2b449323e4434eb242d11ac21488c559e65088705a482cc7b0913" exitCode=0
Nov 03 08:35:38.951334 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:38.950950    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs" event={"ID":"dda9066d-68ff-4e5d-9272-ef793d05228b","Type":"ContainerDied","Data":"34bf3e951da2b449323e4434eb242d11ac21488c559e65088705a482cc7b0913"}
Nov 03 08:35:40.103729 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:40.103702    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs"
Nov 03 08:35:40.227218 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:40.227144    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dda9066d-68ff-4e5d-9272-ef793d05228b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda9066d-68ff-4e5d-9272-ef793d05228b-maas-api-xks-ca\") pod \"dda9066d-68ff-4e5d-9272-ef793d05228b\" (UID: \"dda9066d-68ff-4e5d-9272-ef793d05228b\") "
Nov 03 08:35:40.227375 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:40.227276    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda9066d-68ff-4e5d-9272-ef793d05228b-kube-api-access-2cl6g\" (UniqueName: \"kubernetes.io/projected/dda9066d-68ff-4e5d-9272-ef793d05228b-kube-api-access-2cl6g\") pod \"dda9066d-68ff-4e5d-9272-ef793d05228b\" (UID: \"dda9066d-68ff-4e5d-9272-ef793d05228b\") "
Nov 03 08:35:40.227375 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:40.227298    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda9066d-68ff-4e5d-9272-ef793d05228b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda9066d-68ff-4e5d-9272-ef793d05228b-maas-api-openshift-ca\") pod \"dda9066d-68ff-4e5d-9272-ef793d05228b\" (UID: \"dda9066d-68ff-4e5d-9272-ef793d05228b\") "
Nov 03 08:35:40.227855 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:40.227821    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda9066d-68ff-4e5d-9272-ef793d05228b-maas-api-openshift-ca" pod "dda9066d-68ff-4e5d-9272-ef793d05228b" (UID: "dda9066d-68ff-4e5d-9272-ef793d05228b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:35:40.229255 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:40.229231    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dda9066d-68ff-4e5d-9272-ef793d05228b-maas-api-xks-ca" pod "dda9066d-68ff-4e5d-9272-ef793d05228b" (UID: "dda9066d-68ff-4e5d-9272-ef793d05228b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:35:40.229334 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:40.229259    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda9066d-68ff-4e5d-9272-ef793d05228b-kube-api-access-2cl6g" pod "dda9066d-68ff-4e5d-9272-ef793d05228b" (UID: "dda9066d-68ff-4e5d-9272-ef793d05228b"). InnerVolumeSpecName "kube-api-access-2cl6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:35:40.328736 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:40.328702    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dda9066d-68ff-4e5d-9272-ef793d05228b-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:40.328736 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:40.328732    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cl6g\" (UniqueName: \"kubernetes.io/projected/dda9066d-68ff-4e5d-9272-ef793d05228b-kube-api-access-2cl6g\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:40.328974 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:40.328748    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dda9066d-68ff-4e5d-9272-ef793d05228b-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:40.963687 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:40.963654    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs"
Nov 03 08:35:40.963877 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:40.963684    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs" event={"ID":"dda9066d-68ff-4e5d-9272-ef793d05228b","Type":"ContainerDied","Data":"4d929d9d5a718afe8757422ee4134495ddef6b5af1624ad8813f75353ffaa3e3"}
Nov 03 08:35:40.963877 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:40.963733    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d929d9d5a718afe8757422ee4134495ddef6b5af1624ad8813f75353ffaa3e3"
Nov 03 08:35:47.528359 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:47.528322    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs"]
Nov 03 08:35:47.532207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:47.532183    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0af2f6-fdzxs"]
Nov 03 08:35:48.024231 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:48.024198    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda9066d-68ff-4e5d-9272-ef793d05228b" path="/var/lib/kubelet/pods/dda9066d-68ff-4e5d-9272-ef793d05228b/volumes"
Nov 03 08:35:48.986456 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:48.986416    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:35:52.051264 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:35:52.051227    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:35:55.130895 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:55.130855    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2"]
Nov 03 08:35:55.131315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:55.131274    5070 cpu_manager.go:414] "containerMap: removing container" podUID="dda9066d-68ff-4e5d-9272-ef793d05228b" containerName="revoke-keys"
Nov 03 08:35:55.131315 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:55.131285    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda9066d-68ff-4e5d-9272-ef793d05228b" containerName="revoke-keys"
Nov 03 08:35:55.132106 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:55.131367    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda9066d-68ff-4e5d-9272-ef793d05228b" containerName="revoke-keys"
Nov 03 08:35:55.134593 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:55.134566    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2"
Nov 03 08:35:55.141448 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:55.141425    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:35:55.141616 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:55.141532    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:35:55.141730 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:55.141632    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:35:55.141730 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:55.141699    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:35:55.143345 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:55.143321    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2"]
Nov 03 08:35:55.264907 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:55.264880    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgblk\" (UniqueName: \"kubernetes.io/projected/5beee8d5-0b63-4bb7-853d-e5f606f1228a-kube-api-access-sgblk\") pod \"maas-api-revoke-sub-3d4156f1-c2cx2\" (UID: \"5beee8d5-0b63-4bb7-853d-e5f606f1228a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2"
Nov 03 08:35:55.265037 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:55.264957    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5beee8d5-0b63-4bb7-853d-e5f606f1228a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-c2cx2\" (UID: \"5beee8d5-0b63-4bb7-853d-e5f606f1228a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2"
Nov 03 08:35:55.265037 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:55.264984    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5beee8d5-0b63-4bb7-853d-e5f606f1228a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-c2cx2\" (UID: \"5beee8d5-0b63-4bb7-853d-e5f606f1228a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2"
Nov 03 08:35:55.365688 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:55.365662    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5beee8d5-0b63-4bb7-853d-e5f606f1228a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-c2cx2\" (UID: \"5beee8d5-0b63-4bb7-853d-e5f606f1228a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2"
Nov 03 08:35:55.365825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:55.365699    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5beee8d5-0b63-4bb7-853d-e5f606f1228a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-c2cx2\" (UID: \"5beee8d5-0b63-4bb7-853d-e5f606f1228a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2"
Nov 03 08:35:55.365825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:55.365736    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgblk\" (UniqueName: \"kubernetes.io/projected/5beee8d5-0b63-4bb7-853d-e5f606f1228a-kube-api-access-sgblk\") pod \"maas-api-revoke-sub-3d4156f1-c2cx2\" (UID: \"5beee8d5-0b63-4bb7-853d-e5f606f1228a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2"
Nov 03 08:35:55.366270 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:55.366248    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5beee8d5-0b63-4bb7-853d-e5f606f1228a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-c2cx2\" (UID: \"5beee8d5-0b63-4bb7-853d-e5f606f1228a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2"
Nov 03 08:35:55.368211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:55.368185    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5beee8d5-0b63-4bb7-853d-e5f606f1228a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-c2cx2\" (UID: \"5beee8d5-0b63-4bb7-853d-e5f606f1228a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2"
Nov 03 08:35:55.373492 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:55.373468    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgblk\" (UniqueName: \"kubernetes.io/projected/5beee8d5-0b63-4bb7-853d-e5f606f1228a-kube-api-access-sgblk\") pod \"maas-api-revoke-sub-3d4156f1-c2cx2\" (UID: \"5beee8d5-0b63-4bb7-853d-e5f606f1228a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2"
Nov 03 08:35:55.449262 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:55.449215    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2"
Nov 03 08:35:55.722481 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:55.722453    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2"]
Nov 03 08:35:55.725009 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:35:55.724983    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5beee8d5_0b63_4bb7_853d_e5f606f1228a.slice/crio-3b31aecf1b676ba1ce0afb70e52d7ec1a5bf9ef7518d16722fbe13ec5a6d0293 WatchSource:0}: Error finding container 3b31aecf1b676ba1ce0afb70e52d7ec1a5bf9ef7518d16722fbe13ec5a6d0293: Status 404 returned error can't find the container with id 3b31aecf1b676ba1ce0afb70e52d7ec1a5bf9ef7518d16722fbe13ec5a6d0293
Nov 03 08:35:56.042887 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:56.042831    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2" event={"ID":"5beee8d5-0b63-4bb7-853d-e5f606f1228a","Type":"ContainerStarted","Data":"95caae31e3e997b954b4c78abe9a7064570b8474691517aaff49df4d021e3a88"}
Nov 03 08:35:56.042887 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:56.042876    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2" event={"ID":"5beee8d5-0b63-4bb7-853d-e5f606f1228a","Type":"ContainerStarted","Data":"3b31aecf1b676ba1ce0afb70e52d7ec1a5bf9ef7518d16722fbe13ec5a6d0293"}
Nov 03 08:35:56.059329 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:56.059282    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2" podStartSLOduration=1.059269037 podStartE2EDuration="1.059269037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:35:56.056568796 +0000 UTC m=+2554.171032726" watchObservedRunningTime="2026-11-03 08:35:56.059269037 +0000 UTC m=+2554.173732968"
Nov 03 08:35:57.051505 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:57.051457    5070 generic.go:356] "Generic (PLEG): container finished" podID="5beee8d5-0b63-4bb7-853d-e5f606f1228a" containerID="95caae31e3e997b954b4c78abe9a7064570b8474691517aaff49df4d021e3a88" exitCode=0
Nov 03 08:35:57.051953 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:57.051531    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2" event={"ID":"5beee8d5-0b63-4bb7-853d-e5f606f1228a","Type":"ContainerDied","Data":"95caae31e3e997b954b4c78abe9a7064570b8474691517aaff49df4d021e3a88"}
Nov 03 08:35:58.191884 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:58.191861    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2"
Nov 03 08:35:58.293650 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:58.293621    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5beee8d5-0b63-4bb7-853d-e5f606f1228a-kube-api-access-sgblk\" (UniqueName: \"kubernetes.io/projected/5beee8d5-0b63-4bb7-853d-e5f606f1228a-kube-api-access-sgblk\") pod \"5beee8d5-0b63-4bb7-853d-e5f606f1228a\" (UID: \"5beee8d5-0b63-4bb7-853d-e5f606f1228a\") "
Nov 03 08:35:58.293868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:58.293691    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5beee8d5-0b63-4bb7-853d-e5f606f1228a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5beee8d5-0b63-4bb7-853d-e5f606f1228a-maas-api-openshift-ca\") pod \"5beee8d5-0b63-4bb7-853d-e5f606f1228a\" (UID: \"5beee8d5-0b63-4bb7-853d-e5f606f1228a\") "
Nov 03 08:35:58.293868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:58.293717    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5beee8d5-0b63-4bb7-853d-e5f606f1228a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5beee8d5-0b63-4bb7-853d-e5f606f1228a-maas-api-xks-ca\") pod \"5beee8d5-0b63-4bb7-853d-e5f606f1228a\" (UID: \"5beee8d5-0b63-4bb7-853d-e5f606f1228a\") "
Nov 03 08:35:58.294132 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:58.294105    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5beee8d5-0b63-4bb7-853d-e5f606f1228a-maas-api-openshift-ca" pod "5beee8d5-0b63-4bb7-853d-e5f606f1228a" (UID: "5beee8d5-0b63-4bb7-853d-e5f606f1228a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:35:58.295829 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:58.295798    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5beee8d5-0b63-4bb7-853d-e5f606f1228a-maas-api-xks-ca" pod "5beee8d5-0b63-4bb7-853d-e5f606f1228a" (UID: "5beee8d5-0b63-4bb7-853d-e5f606f1228a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:35:58.295931 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:58.295895    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5beee8d5-0b63-4bb7-853d-e5f606f1228a-kube-api-access-sgblk" pod "5beee8d5-0b63-4bb7-853d-e5f606f1228a" (UID: "5beee8d5-0b63-4bb7-853d-e5f606f1228a"). InnerVolumeSpecName "kube-api-access-sgblk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:35:58.395282 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:58.395253    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgblk\" (UniqueName: \"kubernetes.io/projected/5beee8d5-0b63-4bb7-853d-e5f606f1228a-kube-api-access-sgblk\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:58.395282 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:58.395280    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5beee8d5-0b63-4bb7-853d-e5f606f1228a-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:58.395460 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:58.395292    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5beee8d5-0b63-4bb7-853d-e5f606f1228a-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:35:59.065066 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:59.065031    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2" event={"ID":"5beee8d5-0b63-4bb7-853d-e5f606f1228a","Type":"ContainerDied","Data":"3b31aecf1b676ba1ce0afb70e52d7ec1a5bf9ef7518d16722fbe13ec5a6d0293"}
Nov 03 08:35:59.065247 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:59.065069    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2"
Nov 03 08:35:59.065247 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:35:59.065078    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b31aecf1b676ba1ce0afb70e52d7ec1a5bf9ef7518d16722fbe13ec5a6d0293"
Nov 03 08:36:06.609612 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:36:06.609534    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:36:11.401011 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:36:11.400974    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:36:22.051289 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:36:22.051241    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:36:52.050480 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:36:52.050438    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:37:22.049909 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:37:22.049874    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:37:36.407868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:37:36.407788    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:37:51.212806 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:37:51.212772    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:37:52.050025 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:37:52.049992    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:37:55.572319 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:37:55.572283    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:38:12.003192 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:38:12.003159    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:38:19.304694 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:38:19.304656    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:38:22.050174 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:38:22.050139    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:38:22.688125 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:38:22.688096    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/3.log"
Nov 03 08:38:22.692420 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:38:22.692396    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/3.log"
Nov 03 08:38:22.696638 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:38:22.696608    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-n28db_5738916f-8202-40ed-823e-90f7c9cc4ec9/csi-driver/0.log"
Nov 03 08:38:22.700636 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:38:22.700619    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-n28db_5738916f-8202-40ed-823e-90f7c9cc4ec9/csi-driver/0.log"
Nov 03 08:38:34.814536 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:38:34.814500    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:38:50.491389 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:38:50.491353    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:38:52.051563 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:38:52.051529    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:39:02.909305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:02.909274    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq"]
Nov 03 08:39:02.909868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:02.909686    5070 cpu_manager.go:414] "containerMap: removing container" podUID="5beee8d5-0b63-4bb7-853d-e5f606f1228a" containerName="revoke-keys"
Nov 03 08:39:02.909868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:02.909697    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5beee8d5-0b63-4bb7-853d-e5f606f1228a" containerName="revoke-keys"
Nov 03 08:39:02.909868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:02.909812    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="5beee8d5-0b63-4bb7-853d-e5f606f1228a" containerName="revoke-keys"
Nov 03 08:39:02.913162 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:02.913146    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq"
Nov 03 08:39:02.917630 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:02.917609    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:39:02.917630 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:02.917620    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:39:02.917895 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:02.917768    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:39:02.917978 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:02.917923    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:39:02.922463 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:02.921385    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq"]
Nov 03 08:39:03.031886 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:03.031806    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da7af76a-f748-4e11-b6b6-2555511651cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ad114a5-mv5vq\" (UID: \"da7af76a-f748-4e11-b6b6-2555511651cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq"
Nov 03 08:39:03.031886 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:03.031866    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db4xh\" (UniqueName: \"kubernetes.io/projected/da7af76a-f748-4e11-b6b6-2555511651cc-kube-api-access-db4xh\") pod \"maas-api-revoke-sub-3ad114a5-mv5vq\" (UID: \"da7af76a-f748-4e11-b6b6-2555511651cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq"
Nov 03 08:39:03.032091 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:03.031995    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da7af76a-f748-4e11-b6b6-2555511651cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ad114a5-mv5vq\" (UID: \"da7af76a-f748-4e11-b6b6-2555511651cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq"
Nov 03 08:39:03.133021 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:03.132992    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db4xh\" (UniqueName: \"kubernetes.io/projected/da7af76a-f748-4e11-b6b6-2555511651cc-kube-api-access-db4xh\") pod \"maas-api-revoke-sub-3ad114a5-mv5vq\" (UID: \"da7af76a-f748-4e11-b6b6-2555511651cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq"
Nov 03 08:39:03.133183 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:03.133142    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da7af76a-f748-4e11-b6b6-2555511651cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ad114a5-mv5vq\" (UID: \"da7af76a-f748-4e11-b6b6-2555511651cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq"
Nov 03 08:39:03.133333 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:03.133314    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da7af76a-f748-4e11-b6b6-2555511651cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ad114a5-mv5vq\" (UID: \"da7af76a-f748-4e11-b6b6-2555511651cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq"
Nov 03 08:39:03.133814 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:03.133794    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da7af76a-f748-4e11-b6b6-2555511651cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ad114a5-mv5vq\" (UID: \"da7af76a-f748-4e11-b6b6-2555511651cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq"
Nov 03 08:39:03.135427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:03.135407    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da7af76a-f748-4e11-b6b6-2555511651cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ad114a5-mv5vq\" (UID: \"da7af76a-f748-4e11-b6b6-2555511651cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq"
Nov 03 08:39:03.140346 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:03.140325    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db4xh\" (UniqueName: \"kubernetes.io/projected/da7af76a-f748-4e11-b6b6-2555511651cc-kube-api-access-db4xh\") pod \"maas-api-revoke-sub-3ad114a5-mv5vq\" (UID: \"da7af76a-f748-4e11-b6b6-2555511651cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq"
Nov 03 08:39:03.229197 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:03.229145    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq"
Nov 03 08:39:03.505268 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:03.505242    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq"]
Nov 03 08:39:03.507558 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:39:03.507527    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda7af76a_f748_4e11_b6b6_2555511651cc.slice/crio-9cef1bc02fc6d4e60cc2da69aa3f69a8496a24ac6764ae4aea0510ae03e36f4a WatchSource:0}: Error finding container 9cef1bc02fc6d4e60cc2da69aa3f69a8496a24ac6764ae4aea0510ae03e36f4a: Status 404 returned error can't find the container with id 9cef1bc02fc6d4e60cc2da69aa3f69a8496a24ac6764ae4aea0510ae03e36f4a
Nov 03 08:39:04.105997 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:04.105952    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq" event={"ID":"da7af76a-f748-4e11-b6b6-2555511651cc","Type":"ContainerStarted","Data":"da17196c8656cd1679c7b191c3b30911ef3d0938254efee389a1c428fc4cc60b"}
Nov 03 08:39:04.106379 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:04.106023    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq" event={"ID":"da7af76a-f748-4e11-b6b6-2555511651cc","Type":"ContainerStarted","Data":"9cef1bc02fc6d4e60cc2da69aa3f69a8496a24ac6764ae4aea0510ae03e36f4a"}
Nov 03 08:39:05.113031 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:05.113001    5070 generic.go:356] "Generic (PLEG): container finished" podID="da7af76a-f748-4e11-b6b6-2555511651cc" containerID="da17196c8656cd1679c7b191c3b30911ef3d0938254efee389a1c428fc4cc60b" exitCode=0
Nov 03 08:39:05.113472 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:05.113089    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq" event={"ID":"da7af76a-f748-4e11-b6b6-2555511651cc","Type":"ContainerDied","Data":"da17196c8656cd1679c7b191c3b30911ef3d0938254efee389a1c428fc4cc60b"}
Nov 03 08:39:06.265017 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:06.264988    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq"
Nov 03 08:39:06.463159 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:06.463087    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da7af76a-f748-4e11-b6b6-2555511651cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da7af76a-f748-4e11-b6b6-2555511651cc-maas-api-xks-ca\") pod \"da7af76a-f748-4e11-b6b6-2555511651cc\" (UID: \"da7af76a-f748-4e11-b6b6-2555511651cc\") "
Nov 03 08:39:06.463159 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:06.463134    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da7af76a-f748-4e11-b6b6-2555511651cc-kube-api-access-db4xh\" (UniqueName: \"kubernetes.io/projected/da7af76a-f748-4e11-b6b6-2555511651cc-kube-api-access-db4xh\") pod \"da7af76a-f748-4e11-b6b6-2555511651cc\" (UID: \"da7af76a-f748-4e11-b6b6-2555511651cc\") "
Nov 03 08:39:06.463357 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:06.463174    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da7af76a-f748-4e11-b6b6-2555511651cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da7af76a-f748-4e11-b6b6-2555511651cc-maas-api-openshift-ca\") pod \"da7af76a-f748-4e11-b6b6-2555511651cc\" (UID: \"da7af76a-f748-4e11-b6b6-2555511651cc\") "
Nov 03 08:39:06.463615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:06.463590    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da7af76a-f748-4e11-b6b6-2555511651cc-maas-api-openshift-ca" pod "da7af76a-f748-4e11-b6b6-2555511651cc" (UID: "da7af76a-f748-4e11-b6b6-2555511651cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:39:06.465289 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:06.465264    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da7af76a-f748-4e11-b6b6-2555511651cc-maas-api-xks-ca" pod "da7af76a-f748-4e11-b6b6-2555511651cc" (UID: "da7af76a-f748-4e11-b6b6-2555511651cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:39:06.465368 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:06.465332    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da7af76a-f748-4e11-b6b6-2555511651cc-kube-api-access-db4xh" pod "da7af76a-f748-4e11-b6b6-2555511651cc" (UID: "da7af76a-f748-4e11-b6b6-2555511651cc"). InnerVolumeSpecName "kube-api-access-db4xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:39:06.564669 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:06.564640    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db4xh\" (UniqueName: \"kubernetes.io/projected/da7af76a-f748-4e11-b6b6-2555511651cc-kube-api-access-db4xh\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:06.564669 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:06.564665    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da7af76a-f748-4e11-b6b6-2555511651cc-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:06.564669 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:06.564675    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da7af76a-f748-4e11-b6b6-2555511651cc-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:07.123373 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:07.123346    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq"
Nov 03 08:39:07.123551 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:07.123381    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq" event={"ID":"da7af76a-f748-4e11-b6b6-2555511651cc","Type":"ContainerDied","Data":"9cef1bc02fc6d4e60cc2da69aa3f69a8496a24ac6764ae4aea0510ae03e36f4a"}
Nov 03 08:39:07.123551 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:07.123447    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cef1bc02fc6d4e60cc2da69aa3f69a8496a24ac6764ae4aea0510ae03e36f4a"
Nov 03 08:39:13.716786 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:13.716722    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6"]
Nov 03 08:39:13.717264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:13.717194    5070 cpu_manager.go:414] "containerMap: removing container" podUID="da7af76a-f748-4e11-b6b6-2555511651cc" containerName="revoke-keys"
Nov 03 08:39:13.717264 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:13.717206    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da7af76a-f748-4e11-b6b6-2555511651cc" containerName="revoke-keys"
Nov 03 08:39:13.717343 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:13.717325    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="da7af76a-f748-4e11-b6b6-2555511651cc" containerName="revoke-keys"
Nov 03 08:39:13.720728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:13.720711    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6"
Nov 03 08:39:13.724810 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:13.724771    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:39:13.724810 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:13.724805    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:39:13.725015 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:13.724859    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:39:13.725015 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:13.724939    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:39:13.727955 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:13.727934    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6"]
Nov 03 08:39:13.825110 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:13.825083    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2vsc\" (UniqueName: \"kubernetes.io/projected/de4a1ec7-2092-47c2-9a24-4080a8ac9071-kube-api-access-l2vsc\") pod \"maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6\" (UID: \"de4a1ec7-2092-47c2-9a24-4080a8ac9071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6"
Nov 03 08:39:13.825224 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:13.825167    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de4a1ec7-2092-47c2-9a24-4080a8ac9071-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6\" (UID: \"de4a1ec7-2092-47c2-9a24-4080a8ac9071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6"
Nov 03 08:39:13.825224 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:13.825208    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de4a1ec7-2092-47c2-9a24-4080a8ac9071-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6\" (UID: \"de4a1ec7-2092-47c2-9a24-4080a8ac9071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6"
Nov 03 08:39:13.901704 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:13.901676    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:39:13.930204 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:13.930176    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de4a1ec7-2092-47c2-9a24-4080a8ac9071-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6\" (UID: \"de4a1ec7-2092-47c2-9a24-4080a8ac9071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6"
Nov 03 08:39:13.930335 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:13.930228    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de4a1ec7-2092-47c2-9a24-4080a8ac9071-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6\" (UID: \"de4a1ec7-2092-47c2-9a24-4080a8ac9071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6"
Nov 03 08:39:13.930335 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:13.930266    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2vsc\" (UniqueName: \"kubernetes.io/projected/de4a1ec7-2092-47c2-9a24-4080a8ac9071-kube-api-access-l2vsc\") pod \"maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6\" (UID: \"de4a1ec7-2092-47c2-9a24-4080a8ac9071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6"
Nov 03 08:39:13.931184 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:13.931158    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de4a1ec7-2092-47c2-9a24-4080a8ac9071-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6\" (UID: \"de4a1ec7-2092-47c2-9a24-4080a8ac9071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6"
Nov 03 08:39:13.933350 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:13.933318    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de4a1ec7-2092-47c2-9a24-4080a8ac9071-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6\" (UID: \"de4a1ec7-2092-47c2-9a24-4080a8ac9071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6"
Nov 03 08:39:13.936597 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:13.936577    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2vsc\" (UniqueName: \"kubernetes.io/projected/de4a1ec7-2092-47c2-9a24-4080a8ac9071-kube-api-access-l2vsc\") pod \"maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6\" (UID: \"de4a1ec7-2092-47c2-9a24-4080a8ac9071\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6"
Nov 03 08:39:14.039694 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:14.039646    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6"
Nov 03 08:39:14.313964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:14.313923    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6"]
Nov 03 08:39:14.317121 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:39:14.317095    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde4a1ec7_2092_47c2_9a24_4080a8ac9071.slice/crio-bde65cb5f6fb3cceb576f72fd8cf718f7caa1cfb852f002b5878dc0f375d80e3 WatchSource:0}: Error finding container bde65cb5f6fb3cceb576f72fd8cf718f7caa1cfb852f002b5878dc0f375d80e3: Status 404 returned error can't find the container with id bde65cb5f6fb3cceb576f72fd8cf718f7caa1cfb852f002b5878dc0f375d80e3
Nov 03 08:39:15.167738 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:15.167703    5070 generic.go:356] "Generic (PLEG): container finished" podID="de4a1ec7-2092-47c2-9a24-4080a8ac9071" containerID="855f1115690adbd8b54e74d0f3f2cb77e9ac51dff19eb3f92c0ba4db8f385e9a" exitCode=0
Nov 03 08:39:15.168112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:15.167794    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6" event={"ID":"de4a1ec7-2092-47c2-9a24-4080a8ac9071","Type":"ContainerDied","Data":"855f1115690adbd8b54e74d0f3f2cb77e9ac51dff19eb3f92c0ba4db8f385e9a"}
Nov 03 08:39:15.168112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:15.167861    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6" event={"ID":"de4a1ec7-2092-47c2-9a24-4080a8ac9071","Type":"ContainerStarted","Data":"bde65cb5f6fb3cceb576f72fd8cf718f7caa1cfb852f002b5878dc0f375d80e3"}
Nov 03 08:39:16.315931 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:16.315909    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6"
Nov 03 08:39:16.351694 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:16.351667    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de4a1ec7-2092-47c2-9a24-4080a8ac9071-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de4a1ec7-2092-47c2-9a24-4080a8ac9071-maas-api-xks-ca\") pod \"de4a1ec7-2092-47c2-9a24-4080a8ac9071\" (UID: \"de4a1ec7-2092-47c2-9a24-4080a8ac9071\") "
Nov 03 08:39:16.351839 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:16.351809    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de4a1ec7-2092-47c2-9a24-4080a8ac9071-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de4a1ec7-2092-47c2-9a24-4080a8ac9071-maas-api-openshift-ca\") pod \"de4a1ec7-2092-47c2-9a24-4080a8ac9071\" (UID: \"de4a1ec7-2092-47c2-9a24-4080a8ac9071\") "
Nov 03 08:39:16.351887 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:16.351847    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de4a1ec7-2092-47c2-9a24-4080a8ac9071-kube-api-access-l2vsc\" (UniqueName: \"kubernetes.io/projected/de4a1ec7-2092-47c2-9a24-4080a8ac9071-kube-api-access-l2vsc\") pod \"de4a1ec7-2092-47c2-9a24-4080a8ac9071\" (UID: \"de4a1ec7-2092-47c2-9a24-4080a8ac9071\") "
Nov 03 08:39:16.352206 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:16.352179    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de4a1ec7-2092-47c2-9a24-4080a8ac9071-maas-api-openshift-ca" pod "de4a1ec7-2092-47c2-9a24-4080a8ac9071" (UID: "de4a1ec7-2092-47c2-9a24-4080a8ac9071"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:39:16.353789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:16.353747    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de4a1ec7-2092-47c2-9a24-4080a8ac9071-maas-api-xks-ca" pod "de4a1ec7-2092-47c2-9a24-4080a8ac9071" (UID: "de4a1ec7-2092-47c2-9a24-4080a8ac9071"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:39:16.354068 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:16.354041    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4a1ec7-2092-47c2-9a24-4080a8ac9071-kube-api-access-l2vsc" pod "de4a1ec7-2092-47c2-9a24-4080a8ac9071" (UID: "de4a1ec7-2092-47c2-9a24-4080a8ac9071"). InnerVolumeSpecName "kube-api-access-l2vsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:39:16.452909 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:16.452842    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de4a1ec7-2092-47c2-9a24-4080a8ac9071-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:16.452909 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:16.452869    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2vsc\" (UniqueName: \"kubernetes.io/projected/de4a1ec7-2092-47c2-9a24-4080a8ac9071-kube-api-access-l2vsc\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:16.452909 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:16.452883    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de4a1ec7-2092-47c2-9a24-4080a8ac9071-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:39:17.179507 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:17.179475    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6"
Nov 03 08:39:17.179693 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:17.179509    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6" event={"ID":"de4a1ec7-2092-47c2-9a24-4080a8ac9071","Type":"ContainerDied","Data":"bde65cb5f6fb3cceb576f72fd8cf718f7caa1cfb852f002b5878dc0f375d80e3"}
Nov 03 08:39:17.179693 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:17.179560    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bde65cb5f6fb3cceb576f72fd8cf718f7caa1cfb852f002b5878dc0f375d80e3"
Nov 03 08:39:22.050064 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:39:22.050014    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:39:24.326606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:24.326564    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6"]
Nov 03 08:39:24.328813 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:24.328784    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-890648-n48p6"]
Nov 03 08:39:26.000481 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:26.000446    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:39:26.024651 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:26.024617    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de4a1ec7-2092-47c2-9a24-4080a8ac9071" path="/var/lib/kubelet/pods/de4a1ec7-2092-47c2-9a24-4080a8ac9071/volumes"
Nov 03 08:39:37.496220 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:37.496183    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:39:52.058787 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:39:52.058725    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:39:53.644698 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:39:53.644666    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:40:08.451790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:08.451741    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56"]
Nov 03 08:40:08.452271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:08.452193    5070 cpu_manager.go:414] "containerMap: removing container" podUID="de4a1ec7-2092-47c2-9a24-4080a8ac9071" containerName="revoke-keys"
Nov 03 08:40:08.452271 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:08.452205    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4a1ec7-2092-47c2-9a24-4080a8ac9071" containerName="revoke-keys"
Nov 03 08:40:08.452351 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:08.452293    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4a1ec7-2092-47c2-9a24-4080a8ac9071" containerName="revoke-keys"
Nov 03 08:40:08.455521 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:08.455505    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56"
Nov 03 08:40:08.459799 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:08.459746    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:40:08.460262 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:08.460172    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:40:08.460262 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:08.460175    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:40:08.460262 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:08.460222    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:40:08.463112 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:08.463086    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56"]
Nov 03 08:40:08.561568 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:08.561544    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0dd530-62a7-457e-b769-a677c51e79d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-55n56\" (UID: \"2b0dd530-62a7-457e-b769-a677c51e79d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56"
Nov 03 08:40:08.561716 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:08.561577    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0dd530-62a7-457e-b769-a677c51e79d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-55n56\" (UID: \"2b0dd530-62a7-457e-b769-a677c51e79d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56"
Nov 03 08:40:08.561716 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:08.561668    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmnrw\" (UniqueName: \"kubernetes.io/projected/2b0dd530-62a7-457e-b769-a677c51e79d7-kube-api-access-rmnrw\") pod \"maas-api-revoke-sub-452b6d7b-55n56\" (UID: \"2b0dd530-62a7-457e-b769-a677c51e79d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56"
Nov 03 08:40:08.662239 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:08.662209    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmnrw\" (UniqueName: \"kubernetes.io/projected/2b0dd530-62a7-457e-b769-a677c51e79d7-kube-api-access-rmnrw\") pod \"maas-api-revoke-sub-452b6d7b-55n56\" (UID: \"2b0dd530-62a7-457e-b769-a677c51e79d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56"
Nov 03 08:40:08.662358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:08.662285    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0dd530-62a7-457e-b769-a677c51e79d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-55n56\" (UID: \"2b0dd530-62a7-457e-b769-a677c51e79d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56"
Nov 03 08:40:08.662358 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:08.662318    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0dd530-62a7-457e-b769-a677c51e79d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-55n56\" (UID: \"2b0dd530-62a7-457e-b769-a677c51e79d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56"
Nov 03 08:40:08.662917 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:08.662894    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0dd530-62a7-457e-b769-a677c51e79d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-55n56\" (UID: \"2b0dd530-62a7-457e-b769-a677c51e79d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56"
Nov 03 08:40:08.664745 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:08.664722    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0dd530-62a7-457e-b769-a677c51e79d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-55n56\" (UID: \"2b0dd530-62a7-457e-b769-a677c51e79d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56"
Nov 03 08:40:08.669998 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:08.669975    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmnrw\" (UniqueName: \"kubernetes.io/projected/2b0dd530-62a7-457e-b769-a677c51e79d7-kube-api-access-rmnrw\") pod \"maas-api-revoke-sub-452b6d7b-55n56\" (UID: \"2b0dd530-62a7-457e-b769-a677c51e79d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56"
Nov 03 08:40:08.771121 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:08.771067    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56"
Nov 03 08:40:09.004387 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:09.004356    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:40:09.043127 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:09.043068    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56"]
Nov 03 08:40:09.045731 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:40:09.045709    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b0dd530_62a7_457e_b769_a677c51e79d7.slice/crio-17255e4e25a9ae905baad6188df661ac331b569af2298c64402456de606945f1 WatchSource:0}: Error finding container 17255e4e25a9ae905baad6188df661ac331b569af2298c64402456de606945f1: Status 404 returned error can't find the container with id 17255e4e25a9ae905baad6188df661ac331b569af2298c64402456de606945f1
Nov 03 08:40:09.519525 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:09.519488    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56" event={"ID":"2b0dd530-62a7-457e-b769-a677c51e79d7","Type":"ContainerStarted","Data":"cd5f97c2c9cca4acbade74941067be23e1a7764d8599a4f1220f11f9e038f386"}
Nov 03 08:40:09.520048 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:09.519551    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56" event={"ID":"2b0dd530-62a7-457e-b769-a677c51e79d7","Type":"ContainerStarted","Data":"17255e4e25a9ae905baad6188df661ac331b569af2298c64402456de606945f1"}
Nov 03 08:40:09.536453 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:09.536387    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56" podStartSLOduration=1.536366041 podStartE2EDuration="1.536366041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:40:09.533749585 +0000 UTC m=+2807.648213515" watchObservedRunningTime="2026-11-03 08:40:09.536366041 +0000 UTC m=+2807.650829973"
Nov 03 08:40:10.535498 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:10.535458    5070 generic.go:356] "Generic (PLEG): container finished" podID="2b0dd530-62a7-457e-b769-a677c51e79d7" containerID="cd5f97c2c9cca4acbade74941067be23e1a7764d8599a4f1220f11f9e038f386" exitCode=0
Nov 03 08:40:10.536252 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:10.536225    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56" event={"ID":"2b0dd530-62a7-457e-b769-a677c51e79d7","Type":"ContainerDied","Data":"cd5f97c2c9cca4acbade74941067be23e1a7764d8599a4f1220f11f9e038f386"}
Nov 03 08:40:11.932661 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:11.932633    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56"
Nov 03 08:40:12.032026 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:12.031991    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b0dd530-62a7-457e-b769-a677c51e79d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0dd530-62a7-457e-b769-a677c51e79d7-maas-api-xks-ca\") pod \"2b0dd530-62a7-457e-b769-a677c51e79d7\" (UID: \"2b0dd530-62a7-457e-b769-a677c51e79d7\") "
Nov 03 08:40:12.032210 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:12.032103    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b0dd530-62a7-457e-b769-a677c51e79d7-kube-api-access-rmnrw\" (UniqueName: \"kubernetes.io/projected/2b0dd530-62a7-457e-b769-a677c51e79d7-kube-api-access-rmnrw\") pod \"2b0dd530-62a7-457e-b769-a677c51e79d7\" (UID: \"2b0dd530-62a7-457e-b769-a677c51e79d7\") "
Nov 03 08:40:12.032210 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:12.032171    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b0dd530-62a7-457e-b769-a677c51e79d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0dd530-62a7-457e-b769-a677c51e79d7-maas-api-openshift-ca\") pod \"2b0dd530-62a7-457e-b769-a677c51e79d7\" (UID: \"2b0dd530-62a7-457e-b769-a677c51e79d7\") "
Nov 03 08:40:12.033127 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:12.033094    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b0dd530-62a7-457e-b769-a677c51e79d7-maas-api-openshift-ca" pod "2b0dd530-62a7-457e-b769-a677c51e79d7" (UID: "2b0dd530-62a7-457e-b769-a677c51e79d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:40:12.038272 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:12.038246    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b0dd530-62a7-457e-b769-a677c51e79d7-maas-api-xks-ca" pod "2b0dd530-62a7-457e-b769-a677c51e79d7" (UID: "2b0dd530-62a7-457e-b769-a677c51e79d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:40:12.038836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:12.038812    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b0dd530-62a7-457e-b769-a677c51e79d7-kube-api-access-rmnrw" pod "2b0dd530-62a7-457e-b769-a677c51e79d7" (UID: "2b0dd530-62a7-457e-b769-a677c51e79d7"). InnerVolumeSpecName "kube-api-access-rmnrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:40:12.133749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:12.133666    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b0dd530-62a7-457e-b769-a677c51e79d7-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:40:12.133749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:12.133705    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmnrw\" (UniqueName: \"kubernetes.io/projected/2b0dd530-62a7-457e-b769-a677c51e79d7-kube-api-access-rmnrw\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:40:12.133749 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:12.133721    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b0dd530-62a7-457e-b769-a677c51e79d7-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:40:12.549374 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:12.549286    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56" event={"ID":"2b0dd530-62a7-457e-b769-a677c51e79d7","Type":"ContainerDied","Data":"17255e4e25a9ae905baad6188df661ac331b569af2298c64402456de606945f1"}
Nov 03 08:40:12.549374 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:12.549358    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17255e4e25a9ae905baad6188df661ac331b569af2298c64402456de606945f1"
Nov 03 08:40:12.549374 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:12.549306    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56"
Nov 03 08:40:20.709244 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:20.709211    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:40:22.054341 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:40:22.054300    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:40:22.084336 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.084298    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl"]
Nov 03 08:40:22.085006 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.084980    5070 cpu_manager.go:414] "containerMap: removing container" podUID="2b0dd530-62a7-457e-b769-a677c51e79d7" containerName="revoke-keys"
Nov 03 08:40:22.085006 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.085007    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b0dd530-62a7-457e-b769-a677c51e79d7" containerName="revoke-keys"
Nov 03 08:40:22.085192 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.085163    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b0dd530-62a7-457e-b769-a677c51e79d7" containerName="revoke-keys"
Nov 03 08:40:22.088868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.088846    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl"
Nov 03 08:40:22.096231 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.093094    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:40:22.096231 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.093163    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:40:22.096231 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.093253    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:40:22.097541 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.097504    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl"]
Nov 03 08:40:22.103416 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.097683    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:40:22.120873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.120349    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ab565e-cc66-4c3e-a7b0-27ed4880d00d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-94lwl\" (UID: \"88ab565e-cc66-4c3e-a7b0-27ed4880d00d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl"
Nov 03 08:40:22.120873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.120477    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ab565e-cc66-4c3e-a7b0-27ed4880d00d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-94lwl\" (UID: \"88ab565e-cc66-4c3e-a7b0-27ed4880d00d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl"
Nov 03 08:40:22.120873 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.120571    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnnzs\" (UniqueName: \"kubernetes.io/projected/88ab565e-cc66-4c3e-a7b0-27ed4880d00d-kube-api-access-wnnzs\") pod \"maas-api-revoke-sub-3e31a506-94lwl\" (UID: \"88ab565e-cc66-4c3e-a7b0-27ed4880d00d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl"
Nov 03 08:40:22.222640 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.222596    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ab565e-cc66-4c3e-a7b0-27ed4880d00d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-94lwl\" (UID: \"88ab565e-cc66-4c3e-a7b0-27ed4880d00d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl"
Nov 03 08:40:22.222990 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.222955    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnnzs\" (UniqueName: \"kubernetes.io/projected/88ab565e-cc66-4c3e-a7b0-27ed4880d00d-kube-api-access-wnnzs\") pod \"maas-api-revoke-sub-3e31a506-94lwl\" (UID: \"88ab565e-cc66-4c3e-a7b0-27ed4880d00d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl"
Nov 03 08:40:22.223230 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.223209    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ab565e-cc66-4c3e-a7b0-27ed4880d00d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-94lwl\" (UID: \"88ab565e-cc66-4c3e-a7b0-27ed4880d00d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl"
Nov 03 08:40:22.226075 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.226033    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:40:22.226323 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.226305    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:40:22.231059 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.231037    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:40:22.237370 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.237338    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ab565e-cc66-4c3e-a7b0-27ed4880d00d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-94lwl\" (UID: \"88ab565e-cc66-4c3e-a7b0-27ed4880d00d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl"
Nov 03 08:40:22.251500 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.251451    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnnzs\" (UniqueName: \"kubernetes.io/projected/88ab565e-cc66-4c3e-a7b0-27ed4880d00d-kube-api-access-wnnzs\") pod \"maas-api-revoke-sub-3e31a506-94lwl\" (UID: \"88ab565e-cc66-4c3e-a7b0-27ed4880d00d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl"
Nov 03 08:40:22.254825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.254270    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ab565e-cc66-4c3e-a7b0-27ed4880d00d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-94lwl\" (UID: \"88ab565e-cc66-4c3e-a7b0-27ed4880d00d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl"
Nov 03 08:40:22.424820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.424791    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:40:22.433225 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.433195    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl"
Nov 03 08:40:22.832811 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:22.785639    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl"]
Nov 03 08:40:22.844719 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:40:22.844678    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ab565e_cc66_4c3e_a7b0_27ed4880d00d.slice/crio-764f337d090dc8c9ddb38cd7113270a9cad33fe45b994e4dc0f9db8492e45be5 WatchSource:0}: Error finding container 764f337d090dc8c9ddb38cd7113270a9cad33fe45b994e4dc0f9db8492e45be5: Status 404 returned error can't find the container with id 764f337d090dc8c9ddb38cd7113270a9cad33fe45b994e4dc0f9db8492e45be5
Nov 03 08:40:23.655564 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:23.655523    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl" event={"ID":"88ab565e-cc66-4c3e-a7b0-27ed4880d00d","Type":"ContainerStarted","Data":"38be20d8719e0835b6e357d7a242ead733bd2f18a09c4dee05115f943005fb9a"}
Nov 03 08:40:23.656091 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:23.655590    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl" event={"ID":"88ab565e-cc66-4c3e-a7b0-27ed4880d00d","Type":"ContainerStarted","Data":"764f337d090dc8c9ddb38cd7113270a9cad33fe45b994e4dc0f9db8492e45be5"}
Nov 03 08:40:23.691790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:23.691704    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl" podStartSLOduration=1.6916844690000001 podStartE2EDuration="1.691684469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:40:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:40:23.689114721 +0000 UTC m=+2821.803578652" watchObservedRunningTime="2026-11-03 08:40:23.691684469 +0000 UTC m=+2821.806148401"
Nov 03 08:40:24.663709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:24.663674    5070 generic.go:356] "Generic (PLEG): container finished" podID="88ab565e-cc66-4c3e-a7b0-27ed4880d00d" containerID="38be20d8719e0835b6e357d7a242ead733bd2f18a09c4dee05115f943005fb9a" exitCode=0
Nov 03 08:40:24.663709 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:24.663713    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl" event={"ID":"88ab565e-cc66-4c3e-a7b0-27ed4880d00d","Type":"ContainerDied","Data":"38be20d8719e0835b6e357d7a242ead733bd2f18a09c4dee05115f943005fb9a"}
Nov 03 08:40:25.836899 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:25.836876    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl"
Nov 03 08:40:25.868073 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:25.868044    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88ab565e-cc66-4c3e-a7b0-27ed4880d00d-kube-api-access-wnnzs\" (UniqueName: \"kubernetes.io/projected/88ab565e-cc66-4c3e-a7b0-27ed4880d00d-kube-api-access-wnnzs\") pod \"88ab565e-cc66-4c3e-a7b0-27ed4880d00d\" (UID: \"88ab565e-cc66-4c3e-a7b0-27ed4880d00d\") "
Nov 03 08:40:25.868230 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:25.868138    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88ab565e-cc66-4c3e-a7b0-27ed4880d00d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ab565e-cc66-4c3e-a7b0-27ed4880d00d-maas-api-openshift-ca\") pod \"88ab565e-cc66-4c3e-a7b0-27ed4880d00d\" (UID: \"88ab565e-cc66-4c3e-a7b0-27ed4880d00d\") "
Nov 03 08:40:25.868230 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:25.868183    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88ab565e-cc66-4c3e-a7b0-27ed4880d00d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ab565e-cc66-4c3e-a7b0-27ed4880d00d-maas-api-xks-ca\") pod \"88ab565e-cc66-4c3e-a7b0-27ed4880d00d\" (UID: \"88ab565e-cc66-4c3e-a7b0-27ed4880d00d\") "
Nov 03 08:40:25.868803 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:25.868775    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88ab565e-cc66-4c3e-a7b0-27ed4880d00d-maas-api-openshift-ca" pod "88ab565e-cc66-4c3e-a7b0-27ed4880d00d" (UID: "88ab565e-cc66-4c3e-a7b0-27ed4880d00d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:40:25.871203 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:25.871178    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88ab565e-cc66-4c3e-a7b0-27ed4880d00d-maas-api-xks-ca" pod "88ab565e-cc66-4c3e-a7b0-27ed4880d00d" (UID: "88ab565e-cc66-4c3e-a7b0-27ed4880d00d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:40:25.871802 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:25.871737    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ab565e-cc66-4c3e-a7b0-27ed4880d00d-kube-api-access-wnnzs" pod "88ab565e-cc66-4c3e-a7b0-27ed4880d00d" (UID: "88ab565e-cc66-4c3e-a7b0-27ed4880d00d"). InnerVolumeSpecName "kube-api-access-wnnzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:40:25.972448 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:25.972308    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnnzs\" (UniqueName: \"kubernetes.io/projected/88ab565e-cc66-4c3e-a7b0-27ed4880d00d-kube-api-access-wnnzs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:40:25.972448 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:25.972347    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88ab565e-cc66-4c3e-a7b0-27ed4880d00d-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:40:25.972448 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:25.972367    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88ab565e-cc66-4c3e-a7b0-27ed4880d00d-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:40:26.709782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:26.709276    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl" event={"ID":"88ab565e-cc66-4c3e-a7b0-27ed4880d00d","Type":"ContainerDied","Data":"764f337d090dc8c9ddb38cd7113270a9cad33fe45b994e4dc0f9db8492e45be5"}
Nov 03 08:40:26.709782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:26.709342    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="764f337d090dc8c9ddb38cd7113270a9cad33fe45b994e4dc0f9db8492e45be5"
Nov 03 08:40:26.709782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:26.709415    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl"
Nov 03 08:40:26.741424 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:26.741390    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:40:29.022117 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:29.022083    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc"]
Nov 03 08:40:29.024855 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:29.024828    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4412ebe-7jrfc"]
Nov 03 08:40:30.024696 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:30.024662    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1a5d028-62e6-41ce-ab11-aa12f3cdb63a" path="/var/lib/kubelet/pods/b1a5d028-62e6-41ce-ab11-aa12f3cdb63a/volumes"
Nov 03 08:40:33.060874 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:33.060801    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:40:35.253665 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:35.253626    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-ln9hl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:40:35.254120 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:35.253705    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl" podUID="c4c16356-5579-4aac-aec8-c72808dbb1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:40:38.225785 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:38.224840    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:40:52.053232 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:40:52.053188    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:40:53.358293 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:53.358258    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:40:59.021158 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:59.021124    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2"]
Nov 03 08:40:59.023597 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:40:59.023575    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-c2cx2"]
Nov 03 08:41:00.024403 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:00.024364    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5beee8d5-0b63-4bb7-853d-e5f606f1228a" path="/var/lib/kubelet/pods/5beee8d5-0b63-4bb7-853d-e5f606f1228a/volumes"
Nov 03 08:41:01.502822 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.502742    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"]
Nov 03 08:41:01.503362 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.503343    5070 cpu_manager.go:414] "containerMap: removing container" podUID="88ab565e-cc66-4c3e-a7b0-27ed4880d00d" containerName="revoke-keys"
Nov 03 08:41:01.503362 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.503363    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88ab565e-cc66-4c3e-a7b0-27ed4880d00d" containerName="revoke-keys"
Nov 03 08:41:01.503498 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.503483    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="88ab565e-cc66-4c3e-a7b0-27ed4880d00d" containerName="revoke-keys"
Nov 03 08:41:01.507284 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.507265    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.510886 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.510865    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d3aeb93c-openshift-default-dockercfg-5jv8d\""
Nov 03 08:41:01.520929 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.520902    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"]
Nov 03 08:41:01.558076 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.558052    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-data\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.558197 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.558087    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-workload-certs\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.558251 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.558210    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/987066e0-5f95-4406-8d2d-8231134fc8f6-istiod-ca-cert\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.558309 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.558261    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-workload-socket\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.558361 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.558349    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-podinfo\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.558413 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.558379    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-credential-socket\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.558413 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.558400    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-token\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.558539 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.558522    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99jnr\" (UniqueName: \"kubernetes.io/projected/987066e0-5f95-4406-8d2d-8231134fc8f6-kube-api-access-99jnr\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.558588 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.558573    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-envoy\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.659905 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.659871    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/987066e0-5f95-4406-8d2d-8231134fc8f6-istiod-ca-cert\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.660122 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.659921    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-workload-socket\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.660122 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.659949    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-podinfo\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.660122 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.659984    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-credential-socket\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.660122 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.660013    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-token\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.660122 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.660069    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99jnr\" (UniqueName: \"kubernetes.io/projected/987066e0-5f95-4406-8d2d-8231134fc8f6-kube-api-access-99jnr\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.660400 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.660122    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-envoy\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.660400 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.660186    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-data\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.660400 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.660349    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-workload-certs\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.660400 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.660384    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-workload-socket\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.660629 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.660475    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-credential-socket\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.660629 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.660487    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-data\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.660740 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.660650    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/987066e0-5f95-4406-8d2d-8231134fc8f6-istiod-ca-cert\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.660740 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.660656    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-workload-certs\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.662606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.662587    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-envoy\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.662976 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.662955    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-podinfo\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.668219 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.668195    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99jnr\" (UniqueName: \"kubernetes.io/projected/987066e0-5f95-4406-8d2d-8231134fc8f6-kube-api-access-99jnr\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.669028 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.669010    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-token\") pod \"e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") " pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:01.822958 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:01.822876    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:02.112556 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:02.112530    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"]
Nov 03 08:41:02.115851 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:41:02.115813    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987066e0_5f95_4406_8d2d_8231134fc8f6.slice/crio-3282c890d412ab980519531a0c3d3ebd0bdb5b7506718888b38fe21ef379efe5 WatchSource:0}: Error finding container 3282c890d412ab980519531a0c3d3ebd0bdb5b7506718888b38fe21ef379efe5: Status 404 returned error can't find the container with id 3282c890d412ab980519531a0c3d3ebd0bdb5b7506718888b38fe21ef379efe5
Nov 03 08:41:02.117984 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:02.117948    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:41:02.118110 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:02.118024    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:41:02.118110 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:02.118067    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:41:02.925740 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:02.925701    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw" event={"ID":"987066e0-5f95-4406-8d2d-8231134fc8f6","Type":"ContainerStarted","Data":"391a95fb0c8307acb9a68623526593b3448a21afafb77c8e91fa4c778b87cc6d"}
Nov 03 08:41:02.926148 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:02.925778    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw" event={"ID":"987066e0-5f95-4406-8d2d-8231134fc8f6","Type":"ContainerStarted","Data":"3282c890d412ab980519531a0c3d3ebd0bdb5b7506718888b38fe21ef379efe5"}
Nov 03 08:41:02.943415 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:02.943364    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw" podStartSLOduration=1.943347652 podStartE2EDuration="1.943347652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:41:02.941723523 +0000 UTC m=+2861.056187453" watchObservedRunningTime="2026-11-03 08:41:02.943347652 +0000 UTC m=+2861.057811580"
Nov 03 08:41:03.823934 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:03.823899    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:03.828928 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:03.828902    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:03.933280 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:03.933253    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:03.939066 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:03.934646    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:07.616059 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:07.616024    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:41:18.038700 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:18.038666    5070 generic.go:356] "Generic (PLEG): container finished" podID="987066e0-5f95-4406-8d2d-8231134fc8f6" containerID="391a95fb0c8307acb9a68623526593b3448a21afafb77c8e91fa4c778b87cc6d" exitCode=0
Nov 03 08:41:18.038700 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:18.038703    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw" event={"ID":"987066e0-5f95-4406-8d2d-8231134fc8f6","Type":"ContainerDied","Data":"391a95fb0c8307acb9a68623526593b3448a21afafb77c8e91fa4c778b87cc6d"}
Nov 03 08:41:18.039355 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:18.039120    5070 scope.go:122] "RemoveContainer" containerID="391a95fb0c8307acb9a68623526593b3448a21afafb77c8e91fa4c778b87cc6d"
Nov 03 08:41:18.040346 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:18.040317    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:41:18.040418 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:18.040383    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:41:18.040457 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:18.040416    5070 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 08:41:18.933665 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:18.933634    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:19.046257 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:19.046222    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw" event={"ID":"987066e0-5f95-4406-8d2d-8231134fc8f6","Type":"ContainerStarted","Data":"d5487f5b3bec037c376a4b7474560e7c1134aa78dd07a911b763a1a48792062b"}
Nov 03 08:41:19.823236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:19.823197    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:19.824065 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:19.824038    5070 patch_prober.go:29] interesting pod/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Nov 03 08:41:19.824197 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:19.824087    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw" podUID="987066e0-5f95-4406-8d2d-8231134fc8f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Nov 03 08:41:20.108130 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:20.108100    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk"]
Nov 03 08:41:20.111801 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:20.111778    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk"
Nov 03 08:41:20.116618 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:20.116589    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:41:20.116782 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:20.116733    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:41:20.117710 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:20.117673    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:41:20.117880 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:20.117861    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lpxbz\""
Nov 03 08:41:20.120238 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:20.120113    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk"]
Nov 03 08:41:20.193304 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:20.193273    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/083db012-4b21-44a4-9b4e-218a775c7c02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk\" (UID: \"083db012-4b21-44a4-9b4e-218a775c7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk"
Nov 03 08:41:20.193485 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:20.193401    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw2xq\" (UniqueName: \"kubernetes.io/projected/083db012-4b21-44a4-9b4e-218a775c7c02-kube-api-access-fw2xq\") pod \"maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk\" (UID: \"083db012-4b21-44a4-9b4e-218a775c7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk"
Nov 03 08:41:20.193485 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:20.193439    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/083db012-4b21-44a4-9b4e-218a775c7c02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk\" (UID: \"083db012-4b21-44a4-9b4e-218a775c7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk"
Nov 03 08:41:20.294359 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:20.294328    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw2xq\" (UniqueName: \"kubernetes.io/projected/083db012-4b21-44a4-9b4e-218a775c7c02-kube-api-access-fw2xq\") pod \"maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk\" (UID: \"083db012-4b21-44a4-9b4e-218a775c7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk"
Nov 03 08:41:20.294542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:20.294376    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/083db012-4b21-44a4-9b4e-218a775c7c02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk\" (UID: \"083db012-4b21-44a4-9b4e-218a775c7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk"
Nov 03 08:41:20.294542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:20.294433    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/083db012-4b21-44a4-9b4e-218a775c7c02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk\" (UID: \"083db012-4b21-44a4-9b4e-218a775c7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk"
Nov 03 08:41:20.295131 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:20.295101    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/083db012-4b21-44a4-9b4e-218a775c7c02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk\" (UID: \"083db012-4b21-44a4-9b4e-218a775c7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk"
Nov 03 08:41:20.296857 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:20.296836    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/083db012-4b21-44a4-9b4e-218a775c7c02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk\" (UID: \"083db012-4b21-44a4-9b4e-218a775c7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk"
Nov 03 08:41:20.298281 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:41:20.298256    5070 projected.go:196] Error preparing data for projected volume kube-api-access-fw2xq for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 03 08:41:20.298365 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:41:20.298345    5070 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/083db012-4b21-44a4-9b4e-218a775c7c02-kube-api-access-fw2xq podName:083db012-4b21-44a4-9b4e-218a775c7c02 nodeName:}" failed. No retries permitted until 2026-11-03 08:41:20.798317102 +0000 UTC m=+2878.912781025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fw2xq" (UniqueName: "kubernetes.io/projected/083db012-4b21-44a4-9b4e-218a775c7c02-kube-api-access-fw2xq") pod "maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk" (UID: "083db012-4b21-44a4-9b4e-218a775c7c02") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 03 08:41:20.824231 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:20.824189    5070 patch_prober.go:29] interesting pod/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Nov 03 08:41:20.824425 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:20.824249    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw" podUID="987066e0-5f95-4406-8d2d-8231134fc8f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Nov 03 08:41:20.899166 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:20.899133    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw2xq\" (UniqueName: \"kubernetes.io/projected/083db012-4b21-44a4-9b4e-218a775c7c02-kube-api-access-fw2xq\") pod \"maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk\" (UID: \"083db012-4b21-44a4-9b4e-218a775c7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk"
Nov 03 08:41:20.906555 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:20.906532    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw2xq\" (UniqueName: \"kubernetes.io/projected/083db012-4b21-44a4-9b4e-218a775c7c02-kube-api-access-fw2xq\") pod \"maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk\" (UID: \"083db012-4b21-44a4-9b4e-218a775c7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk"
Nov 03 08:41:21.026619 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:21.026583    5070 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk" secret="" err="secret \"maas-api-cleanup-dockercfg-lpxbz\" not found"
Nov 03 08:41:21.026848 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:21.026651    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk"
Nov 03 08:41:21.324345 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:21.324284    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk"]
Nov 03 08:41:21.326882 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:41:21.326851    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083db012_4b21_44a4_9b4e_218a775c7c02.slice/crio-27ad3bd4ca1c3a54f6d9d143031fa0a257f014fd41ffb8dba44060003a7fd56c WatchSource:0}: Error finding container 27ad3bd4ca1c3a54f6d9d143031fa0a257f014fd41ffb8dba44060003a7fd56c: Status 404 returned error can't find the container with id 27ad3bd4ca1c3a54f6d9d143031fa0a257f014fd41ffb8dba44060003a7fd56c
Nov 03 08:41:21.823945 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:21.823915    5070 patch_prober.go:29] interesting pod/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Nov 03 08:41:21.824097 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:21.823962    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw" podUID="987066e0-5f95-4406-8d2d-8231134fc8f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Nov 03 08:41:22.050262 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:41:22.050184    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:41:22.091973 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:22.091943    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk" event={"ID":"083db012-4b21-44a4-9b4e-218a775c7c02","Type":"ContainerStarted","Data":"a2f3b7f5ff3a936825fb41bfd5d4d380f4d3067d764491af08c61e1a1b69769b"}
Nov 03 08:41:22.092150 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:22.092000    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk" event={"ID":"083db012-4b21-44a4-9b4e-218a775c7c02","Type":"ContainerStarted","Data":"27ad3bd4ca1c3a54f6d9d143031fa0a257f014fd41ffb8dba44060003a7fd56c"}
Nov 03 08:41:22.092296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:22.092277    5070 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk" secret="" err="secret \"maas-api-cleanup-dockercfg-lpxbz\" not found"
Nov 03 08:41:22.109320 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:22.109276    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk" podStartSLOduration=2.109263087 podStartE2EDuration="2.109263087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:41:22.106979852 +0000 UTC m=+2880.221443778" watchObservedRunningTime="2026-11-03 08:41:22.109263087 +0000 UTC m=+2880.223727017"
Nov 03 08:41:22.823557 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:22.823465    5070 patch_prober.go:29] interesting pod/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Nov 03 08:41:22.823557 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:22.823543    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw" podUID="987066e0-5f95-4406-8d2d-8231134fc8f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Nov 03 08:41:23.099130 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:23.099051    5070 generic.go:356] "Generic (PLEG): container finished" podID="083db012-4b21-44a4-9b4e-218a775c7c02" containerID="a2f3b7f5ff3a936825fb41bfd5d4d380f4d3067d764491af08c61e1a1b69769b" exitCode=0
Nov 03 08:41:23.099384 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:23.099132    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk" event={"ID":"083db012-4b21-44a4-9b4e-218a775c7c02","Type":"ContainerDied","Data":"a2f3b7f5ff3a936825fb41bfd5d4d380f4d3067d764491af08c61e1a1b69769b"}
Nov 03 08:41:23.824111 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:23.824079    5070 patch_prober.go:29] interesting pod/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Nov 03 08:41:23.824527 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:23.824141    5070 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw" podUID="987066e0-5f95-4406-8d2d-8231134fc8f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Nov 03 08:41:24.249155 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:24.249126    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk"
Nov 03 08:41:24.339309 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:24.339281    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/083db012-4b21-44a4-9b4e-218a775c7c02-kube-api-access-fw2xq\" (UniqueName: \"kubernetes.io/projected/083db012-4b21-44a4-9b4e-218a775c7c02-kube-api-access-fw2xq\") pod \"083db012-4b21-44a4-9b4e-218a775c7c02\" (UID: \"083db012-4b21-44a4-9b4e-218a775c7c02\") "
Nov 03 08:41:24.339454 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:24.339382    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/083db012-4b21-44a4-9b4e-218a775c7c02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/083db012-4b21-44a4-9b4e-218a775c7c02-maas-api-xks-ca\") pod \"083db012-4b21-44a4-9b4e-218a775c7c02\" (UID: \"083db012-4b21-44a4-9b4e-218a775c7c02\") "
Nov 03 08:41:24.339454 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:24.339402    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/083db012-4b21-44a4-9b4e-218a775c7c02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/083db012-4b21-44a4-9b4e-218a775c7c02-maas-api-openshift-ca\") pod \"083db012-4b21-44a4-9b4e-218a775c7c02\" (UID: \"083db012-4b21-44a4-9b4e-218a775c7c02\") "
Nov 03 08:41:24.339883 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:24.339854    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/083db012-4b21-44a4-9b4e-218a775c7c02-maas-api-openshift-ca" pod "083db012-4b21-44a4-9b4e-218a775c7c02" (UID: "083db012-4b21-44a4-9b4e-218a775c7c02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:41:24.341444 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:24.341419    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083db012-4b21-44a4-9b4e-218a775c7c02-kube-api-access-fw2xq" pod "083db012-4b21-44a4-9b4e-218a775c7c02" (UID: "083db012-4b21-44a4-9b4e-218a775c7c02"). InnerVolumeSpecName "kube-api-access-fw2xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:41:24.341549 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:24.341416    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/083db012-4b21-44a4-9b4e-218a775c7c02-maas-api-xks-ca" pod "083db012-4b21-44a4-9b4e-218a775c7c02" (UID: "083db012-4b21-44a4-9b4e-218a775c7c02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:41:24.440600 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:24.440543    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw2xq\" (UniqueName: \"kubernetes.io/projected/083db012-4b21-44a4-9b4e-218a775c7c02-kube-api-access-fw2xq\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:24.440600 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:24.440566    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/083db012-4b21-44a4-9b4e-218a775c7c02-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:24.440600 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:24.440576    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/083db012-4b21-44a4-9b4e-218a775c7c02-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:24.827544 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:24.827467    5070 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:24.827956 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:24.827724    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:24.828478 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:24.828463    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:25.112769 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:25.112733    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk"
Nov 03 08:41:25.112952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:25.112807    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk" event={"ID":"083db012-4b21-44a4-9b4e-218a775c7c02","Type":"ContainerDied","Data":"27ad3bd4ca1c3a54f6d9d143031fa0a257f014fd41ffb8dba44060003a7fd56c"}
Nov 03 08:41:25.112952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:25.112867    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27ad3bd4ca1c3a54f6d9d143031fa0a257f014fd41ffb8dba44060003a7fd56c"
Nov 03 08:41:30.869396 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:30.869361    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk"]
Nov 03 08:41:30.872707 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:30.872680    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3aeb93c-tx6pk"]
Nov 03 08:41:32.026332 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:32.026296    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="083db012-4b21-44a4-9b4e-218a775c7c02" path="/var/lib/kubelet/pods/083db012-4b21-44a4-9b4e-218a775c7c02/volumes"
Nov 03 08:41:32.156394 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:32.156361    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:41:36.407183 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:36.407142    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"]
Nov 03 08:41:36.407699 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:36.407431    5070 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw" podUID="987066e0-5f95-4406-8d2d-8231134fc8f6" containerName="istio-proxy" containerID="cri-o://d5487f5b3bec037c376a4b7474560e7c1134aa78dd07a911b763a1a48792062b" gracePeriod=30
Nov 03 08:41:39.830715 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:39.830677    5070 patch_prober.go:29] interesting pod/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 08:41:39.831132 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:39.830736    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw" podUID="987066e0-5f95-4406-8d2d-8231134fc8f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 08:41:41.675189 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.675133    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:41.805049 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.805024    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-token\" (UniqueName: \"kubernetes.io/projected/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-token\") pod \"987066e0-5f95-4406-8d2d-8231134fc8f6\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") "
Nov 03 08:41:41.805197 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.805072    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-workload-certs\") pod \"987066e0-5f95-4406-8d2d-8231134fc8f6\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") "
Nov 03 08:41:41.805197 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.805111    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-envoy\") pod \"987066e0-5f95-4406-8d2d-8231134fc8f6\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") "
Nov 03 08:41:41.805289 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.805220    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/987066e0-5f95-4406-8d2d-8231134fc8f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/987066e0-5f95-4406-8d2d-8231134fc8f6-istiod-ca-cert\") pod \"987066e0-5f95-4406-8d2d-8231134fc8f6\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") "
Nov 03 08:41:41.805334 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.805317    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/987066e0-5f95-4406-8d2d-8231134fc8f6-kube-api-access-99jnr\" (UniqueName: \"kubernetes.io/projected/987066e0-5f95-4406-8d2d-8231134fc8f6-kube-api-access-99jnr\") pod \"987066e0-5f95-4406-8d2d-8231134fc8f6\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") "
Nov 03 08:41:41.805384 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.805361    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-credential-socket\") pod \"987066e0-5f95-4406-8d2d-8231134fc8f6\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") "
Nov 03 08:41:41.805468 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.805424    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-workload-certs" pod "987066e0-5f95-4406-8d2d-8231134fc8f6" (UID: "987066e0-5f95-4406-8d2d-8231134fc8f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:41:41.805553 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.805465    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-data\") pod \"987066e0-5f95-4406-8d2d-8231134fc8f6\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") "
Nov 03 08:41:41.805553 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.805514    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-workload-socket\") pod \"987066e0-5f95-4406-8d2d-8231134fc8f6\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") "
Nov 03 08:41:41.805665 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.805603    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-podinfo\") pod \"987066e0-5f95-4406-8d2d-8231134fc8f6\" (UID: \"987066e0-5f95-4406-8d2d-8231134fc8f6\") "
Nov 03 08:41:41.805665 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.805619    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-credential-socket" pod "987066e0-5f95-4406-8d2d-8231134fc8f6" (UID: "987066e0-5f95-4406-8d2d-8231134fc8f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:41:41.805665 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.805654    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-data" pod "987066e0-5f95-4406-8d2d-8231134fc8f6" (UID: "987066e0-5f95-4406-8d2d-8231134fc8f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:41:41.805852 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.805681    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987066e0-5f95-4406-8d2d-8231134fc8f6-istiod-ca-cert" pod "987066e0-5f95-4406-8d2d-8231134fc8f6" (UID: "987066e0-5f95-4406-8d2d-8231134fc8f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:41:41.805852 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.805821    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-workload-socket" pod "987066e0-5f95-4406-8d2d-8231134fc8f6" (UID: "987066e0-5f95-4406-8d2d-8231134fc8f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:41:41.806059 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.806032    5070 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-workload-certs\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:41.806129 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.806058    5070 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/987066e0-5f95-4406-8d2d-8231134fc8f6-istiod-ca-cert\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:41.806129 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.806070    5070 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-credential-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:41.806129 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.806079    5070 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-data\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:41.806129 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.806088    5070 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-workload-socket\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:41.807551 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.807531    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987066e0-5f95-4406-8d2d-8231134fc8f6-kube-api-access-99jnr" pod "987066e0-5f95-4406-8d2d-8231134fc8f6" (UID: "987066e0-5f95-4406-8d2d-8231134fc8f6"). InnerVolumeSpecName "kube-api-access-99jnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:41:41.807699 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.807682    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-podinfo" pod "987066e0-5f95-4406-8d2d-8231134fc8f6" (UID: "987066e0-5f95-4406-8d2d-8231134fc8f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 08:41:41.807869 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.807850    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-token" pod "987066e0-5f95-4406-8d2d-8231134fc8f6" (UID: "987066e0-5f95-4406-8d2d-8231134fc8f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:41:41.807921 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.807858    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-envoy" pod "987066e0-5f95-4406-8d2d-8231134fc8f6" (UID: "987066e0-5f95-4406-8d2d-8231134fc8f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 08:41:41.907426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.907398    5070 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-podinfo\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:41.907426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.907422    5070 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-token\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:41.907567 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.907436    5070 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/987066e0-5f95-4406-8d2d-8231134fc8f6-istio-envoy\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:41.907567 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:41.907449    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99jnr\" (UniqueName: \"kubernetes.io/projected/987066e0-5f95-4406-8d2d-8231134fc8f6-kube-api-access-99jnr\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:41:42.209639 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:42.209608    5070 generic.go:356] "Generic (PLEG): container finished" podID="987066e0-5f95-4406-8d2d-8231134fc8f6" containerID="d5487f5b3bec037c376a4b7474560e7c1134aa78dd07a911b763a1a48792062b" exitCode=0
Nov 03 08:41:42.209796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:42.209664    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"
Nov 03 08:41:42.209796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:42.209695    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw" event={"ID":"987066e0-5f95-4406-8d2d-8231134fc8f6","Type":"ContainerDied","Data":"d5487f5b3bec037c376a4b7474560e7c1134aa78dd07a911b763a1a48792062b"}
Nov 03 08:41:42.209796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:42.209749    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw" event={"ID":"987066e0-5f95-4406-8d2d-8231134fc8f6","Type":"ContainerDied","Data":"3282c890d412ab980519531a0c3d3ebd0bdb5b7506718888b38fe21ef379efe5"}
Nov 03 08:41:42.209952 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:42.209799    5070 scope.go:122] "RemoveContainer" containerID="d5487f5b3bec037c376a4b7474560e7c1134aa78dd07a911b763a1a48792062b"
Nov 03 08:41:42.220324 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:42.220306    5070 scope.go:122] "RemoveContainer" containerID="391a95fb0c8307acb9a68623526593b3448a21afafb77c8e91fa4c778b87cc6d"
Nov 03 08:41:42.225151 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:42.225121    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"]
Nov 03 08:41:42.228186 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:42.228164    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d3aeb93c-openshift-default-78694c5469-wfslw"]
Nov 03 08:41:42.231314 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:42.231286    5070 scope.go:122] "RemoveContainer" containerID="d5487f5b3bec037c376a4b7474560e7c1134aa78dd07a911b763a1a48792062b"
Nov 03 08:41:42.231606 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:41:42.231589    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5487f5b3bec037c376a4b7474560e7c1134aa78dd07a911b763a1a48792062b\": container with ID starting with d5487f5b3bec037c376a4b7474560e7c1134aa78dd07a911b763a1a48792062b not found: ID does not exist" containerID="d5487f5b3bec037c376a4b7474560e7c1134aa78dd07a911b763a1a48792062b"
Nov 03 08:41:42.231695 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:42.231618    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5487f5b3bec037c376a4b7474560e7c1134aa78dd07a911b763a1a48792062b"} err="failed to get container status \"d5487f5b3bec037c376a4b7474560e7c1134aa78dd07a911b763a1a48792062b\": rpc error: code = NotFound desc = could not find container \"d5487f5b3bec037c376a4b7474560e7c1134aa78dd07a911b763a1a48792062b\": container with ID starting with d5487f5b3bec037c376a4b7474560e7c1134aa78dd07a911b763a1a48792062b not found: ID does not exist"
Nov 03 08:41:42.231695 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:42.231640    5070 scope.go:122] "RemoveContainer" containerID="391a95fb0c8307acb9a68623526593b3448a21afafb77c8e91fa4c778b87cc6d"
Nov 03 08:41:42.231935 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:41:42.231914    5070 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"391a95fb0c8307acb9a68623526593b3448a21afafb77c8e91fa4c778b87cc6d\": container with ID starting with 391a95fb0c8307acb9a68623526593b3448a21afafb77c8e91fa4c778b87cc6d not found: ID does not exist" containerID="391a95fb0c8307acb9a68623526593b3448a21afafb77c8e91fa4c778b87cc6d"
Nov 03 08:41:42.231976 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:42.231942    5070 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391a95fb0c8307acb9a68623526593b3448a21afafb77c8e91fa4c778b87cc6d"} err="failed to get container status \"391a95fb0c8307acb9a68623526593b3448a21afafb77c8e91fa4c778b87cc6d\": rpc error: code = NotFound desc = could not find container \"391a95fb0c8307acb9a68623526593b3448a21afafb77c8e91fa4c778b87cc6d\": container with ID starting with 391a95fb0c8307acb9a68623526593b3448a21afafb77c8e91fa4c778b87cc6d not found: ID does not exist"
Nov 03 08:41:44.024432 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:44.024391    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="987066e0-5f95-4406-8d2d-8231134fc8f6" path="/var/lib/kubelet/pods/987066e0-5f95-4406-8d2d-8231134fc8f6/volumes"
Nov 03 08:41:44.391854 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:41:44.391821    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:41:52.055104 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:41:52.055044    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:42:15.469941 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:42:15.469856    5070 scope.go:122] "RemoveContainer" containerID="95caae31e3e997b954b4c78abe9a7064570b8474691517aaff49df4d021e3a88"
Nov 03 08:42:15.478498 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:42:15.478480    5070 scope.go:122] "RemoveContainer" containerID="34bf3e951da2b449323e4434eb242d11ac21488c559e65088705a482cc7b0913"
Nov 03 08:42:15.486470 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:42:15.486451    5070 scope.go:122] "RemoveContainer" containerID="f34578bf493c1033cab4b9ba1388acc1ae280717c38c77bbca6fc4a22411126c"
Nov 03 08:42:22.050137 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:42:22.050105    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:42:52.051609 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:42:52.051567    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:43:15.927753 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:15.927719    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp"]
Nov 03 08:43:15.928200 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:15.928183    5070 cpu_manager.go:414] "containerMap: removing container" podUID="987066e0-5f95-4406-8d2d-8231134fc8f6" containerName="istio-proxy"
Nov 03 08:43:15.928200 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:15.928200    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987066e0-5f95-4406-8d2d-8231134fc8f6" containerName="istio-proxy"
Nov 03 08:43:15.928288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:15.928225    5070 cpu_manager.go:414] "containerMap: removing container" podUID="987066e0-5f95-4406-8d2d-8231134fc8f6" containerName="istio-proxy"
Nov 03 08:43:15.928288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:15.928231    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987066e0-5f95-4406-8d2d-8231134fc8f6" containerName="istio-proxy"
Nov 03 08:43:15.928288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:15.928239    5070 cpu_manager.go:414] "containerMap: removing container" podUID="083db012-4b21-44a4-9b4e-218a775c7c02" containerName="revoke-keys"
Nov 03 08:43:15.928288 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:15.928244    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="083db012-4b21-44a4-9b4e-218a775c7c02" containerName="revoke-keys"
Nov 03 08:43:15.928427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:15.928325    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="987066e0-5f95-4406-8d2d-8231134fc8f6" containerName="istio-proxy"
Nov 03 08:43:15.928427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:15.928338    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="987066e0-5f95-4406-8d2d-8231134fc8f6" containerName="istio-proxy"
Nov 03 08:43:15.928427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:15.928345    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="083db012-4b21-44a4-9b4e-218a775c7c02" containerName="revoke-keys"
Nov 03 08:43:15.931359 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:15.931340    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp"
Nov 03 08:43:15.936089 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:15.936069    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:43:15.936240 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:15.936155    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tp6bz\""
Nov 03 08:43:15.939929 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:15.939276    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp"]
Nov 03 08:43:15.940775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:15.940744    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:43:15.940892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:15.940867    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:43:16.029234 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:16.029206    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d03f793-7a8a-4261-b18d-3f32bca927d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp\" (UID: \"6d03f793-7a8a-4261-b18d-3f32bca927d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp"
Nov 03 08:43:16.029383 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:16.029247    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddtck\" (UniqueName: \"kubernetes.io/projected/6d03f793-7a8a-4261-b18d-3f32bca927d8-kube-api-access-ddtck\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp\" (UID: \"6d03f793-7a8a-4261-b18d-3f32bca927d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp"
Nov 03 08:43:16.029383 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:16.029366    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d03f793-7a8a-4261-b18d-3f32bca927d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp\" (UID: \"6d03f793-7a8a-4261-b18d-3f32bca927d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp"
Nov 03 08:43:16.130576 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:16.130550    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d03f793-7a8a-4261-b18d-3f32bca927d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp\" (UID: \"6d03f793-7a8a-4261-b18d-3f32bca927d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp"
Nov 03 08:43:16.130698 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:16.130587    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddtck\" (UniqueName: \"kubernetes.io/projected/6d03f793-7a8a-4261-b18d-3f32bca927d8-kube-api-access-ddtck\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp\" (UID: \"6d03f793-7a8a-4261-b18d-3f32bca927d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp"
Nov 03 08:43:16.130698 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:16.130638    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d03f793-7a8a-4261-b18d-3f32bca927d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp\" (UID: \"6d03f793-7a8a-4261-b18d-3f32bca927d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp"
Nov 03 08:43:16.131211 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:16.131181    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d03f793-7a8a-4261-b18d-3f32bca927d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp\" (UID: \"6d03f793-7a8a-4261-b18d-3f32bca927d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp"
Nov 03 08:43:16.133041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:16.133023    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d03f793-7a8a-4261-b18d-3f32bca927d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp\" (UID: \"6d03f793-7a8a-4261-b18d-3f32bca927d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp"
Nov 03 08:43:16.137907 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:16.137887    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddtck\" (UniqueName: \"kubernetes.io/projected/6d03f793-7a8a-4261-b18d-3f32bca927d8-kube-api-access-ddtck\") pod \"maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp\" (UID: \"6d03f793-7a8a-4261-b18d-3f32bca927d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp"
Nov 03 08:43:16.246611 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:16.246557    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp"
Nov 03 08:43:16.521912 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:16.521891    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp"]
Nov 03 08:43:16.524264 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:43:16.524232    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d03f793_7a8a_4261_b18d_3f32bca927d8.slice/crio-d0fcdad2396a5094ba7f3bde823395b2e0929708f6083f2de36f66365c76d6d9 WatchSource:0}: Error finding container d0fcdad2396a5094ba7f3bde823395b2e0929708f6083f2de36f66365c76d6d9: Status 404 returned error can't find the container with id d0fcdad2396a5094ba7f3bde823395b2e0929708f6083f2de36f66365c76d6d9
Nov 03 08:43:16.744375 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:16.744345    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp" event={"ID":"6d03f793-7a8a-4261-b18d-3f32bca927d8","Type":"ContainerStarted","Data":"d0fcdad2396a5094ba7f3bde823395b2e0929708f6083f2de36f66365c76d6d9"}
Nov 03 08:43:17.751703 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:17.751623    5070 generic.go:356] "Generic (PLEG): container finished" podID="6d03f793-7a8a-4261-b18d-3f32bca927d8" containerID="e056bd1f6b1ec0b215a92f8bd713e7ce66f8dada2c4a1a47b8ac912849c2cac0" exitCode=0
Nov 03 08:43:17.752134 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:17.751711    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp" event={"ID":"6d03f793-7a8a-4261-b18d-3f32bca927d8","Type":"ContainerDied","Data":"e056bd1f6b1ec0b215a92f8bd713e7ce66f8dada2c4a1a47b8ac912849c2cac0"}
Nov 03 08:43:18.901134 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:18.901107    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp"
Nov 03 08:43:18.956546 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:18.956507    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d03f793-7a8a-4261-b18d-3f32bca927d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d03f793-7a8a-4261-b18d-3f32bca927d8-maas-api-openshift-ca\") pod \"6d03f793-7a8a-4261-b18d-3f32bca927d8\" (UID: \"6d03f793-7a8a-4261-b18d-3f32bca927d8\") "
Nov 03 08:43:18.956773 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:18.956552    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d03f793-7a8a-4261-b18d-3f32bca927d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d03f793-7a8a-4261-b18d-3f32bca927d8-maas-api-xks-ca\") pod \"6d03f793-7a8a-4261-b18d-3f32bca927d8\" (UID: \"6d03f793-7a8a-4261-b18d-3f32bca927d8\") "
Nov 03 08:43:18.956773 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:18.956703    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d03f793-7a8a-4261-b18d-3f32bca927d8-kube-api-access-ddtck\" (UniqueName: \"kubernetes.io/projected/6d03f793-7a8a-4261-b18d-3f32bca927d8-kube-api-access-ddtck\") pod \"6d03f793-7a8a-4261-b18d-3f32bca927d8\" (UID: \"6d03f793-7a8a-4261-b18d-3f32bca927d8\") "
Nov 03 08:43:18.957001 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:18.956977    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d03f793-7a8a-4261-b18d-3f32bca927d8-maas-api-openshift-ca" pod "6d03f793-7a8a-4261-b18d-3f32bca927d8" (UID: "6d03f793-7a8a-4261-b18d-3f32bca927d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:43:18.957172 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:18.957151    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d03f793-7a8a-4261-b18d-3f32bca927d8-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:18.959380 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:18.959344    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d03f793-7a8a-4261-b18d-3f32bca927d8-maas-api-xks-ca" pod "6d03f793-7a8a-4261-b18d-3f32bca927d8" (UID: "6d03f793-7a8a-4261-b18d-3f32bca927d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:43:18.959539 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:18.959494    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d03f793-7a8a-4261-b18d-3f32bca927d8-kube-api-access-ddtck" pod "6d03f793-7a8a-4261-b18d-3f32bca927d8" (UID: "6d03f793-7a8a-4261-b18d-3f32bca927d8"). InnerVolumeSpecName "kube-api-access-ddtck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:43:19.057854 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:19.057794    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d03f793-7a8a-4261-b18d-3f32bca927d8-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:19.057854 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:19.057821    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddtck\" (UniqueName: \"kubernetes.io/projected/6d03f793-7a8a-4261-b18d-3f32bca927d8-kube-api-access-ddtck\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:19.764653 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:19.764617    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp" event={"ID":"6d03f793-7a8a-4261-b18d-3f32bca927d8","Type":"ContainerDied","Data":"d0fcdad2396a5094ba7f3bde823395b2e0929708f6083f2de36f66365c76d6d9"}
Nov 03 08:43:19.764857 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:19.764686    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp"
Nov 03 08:43:19.764857 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:19.764697    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0fcdad2396a5094ba7f3bde823395b2e0929708f6083f2de36f66365c76d6d9"
Nov 03 08:43:22.050579 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:43:22.050542    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:43:22.727601 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:22.727574    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/3.log"
Nov 03 08:43:22.732414 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:22.732382    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/3.log"
Nov 03 08:43:22.736391 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:22.736371    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-n28db_5738916f-8202-40ed-823e-90f7c9cc4ec9/csi-driver/0.log"
Nov 03 08:43:22.741111 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:22.741091    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-n28db_5738916f-8202-40ed-823e-90f7c9cc4ec9/csi-driver/0.log"
Nov 03 08:43:31.922458 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:31.922422    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp"]
Nov 03 08:43:31.928363 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:31.928340    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-b34c73-mkrbp"]
Nov 03 08:43:32.023835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:32.023792    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d03f793-7a8a-4261-b18d-3f32bca927d8" path="/var/lib/kubelet/pods/6d03f793-7a8a-4261-b18d-3f32bca927d8/volumes"
Nov 03 08:43:33.272140 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:33.272060    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:43:40.483279 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:40.483233    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:43:45.673305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:45.673274    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:43:50.332792 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:50.332747    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cnqgv"]
Nov 03 08:43:50.333284 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:50.333172    5070 cpu_manager.go:414] "containerMap: removing container" podUID="6d03f793-7a8a-4261-b18d-3f32bca927d8" containerName="revoke-keys"
Nov 03 08:43:50.333284 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:50.333183    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d03f793-7a8a-4261-b18d-3f32bca927d8" containerName="revoke-keys"
Nov 03 08:43:50.333284 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:50.333271    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d03f793-7a8a-4261-b18d-3f32bca927d8" containerName="revoke-keys"
Nov 03 08:43:50.337871 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:50.337853    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cnqgv"
Nov 03 08:43:50.341737 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:50.341703    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:43:50.341977 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:50.341887    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tp6bz\""
Nov 03 08:43:50.341977 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:50.341969    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:43:50.342111 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:50.342022    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:43:50.344396 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:50.344372    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cnqgv"]
Nov 03 08:43:50.427892 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:50.427865    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26506959-bd2e-498f-9962-2ec4e27b4cb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-cnqgv\" (UID: \"26506959-bd2e-498f-9962-2ec4e27b4cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cnqgv"
Nov 03 08:43:50.428013 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:50.427917    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26506959-bd2e-498f-9962-2ec4e27b4cb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-cnqgv\" (UID: \"26506959-bd2e-498f-9962-2ec4e27b4cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cnqgv"
Nov 03 08:43:50.428013 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:50.427935    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8pjg\" (UniqueName: \"kubernetes.io/projected/26506959-bd2e-498f-9962-2ec4e27b4cb5-kube-api-access-z8pjg\") pod \"maas-api-revoke-sub-f03629db-cnqgv\" (UID: \"26506959-bd2e-498f-9962-2ec4e27b4cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cnqgv"
Nov 03 08:43:50.529330 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:50.529303    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8pjg\" (UniqueName: \"kubernetes.io/projected/26506959-bd2e-498f-9962-2ec4e27b4cb5-kube-api-access-z8pjg\") pod \"maas-api-revoke-sub-f03629db-cnqgv\" (UID: \"26506959-bd2e-498f-9962-2ec4e27b4cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cnqgv"
Nov 03 08:43:50.529446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:50.529375    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26506959-bd2e-498f-9962-2ec4e27b4cb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-cnqgv\" (UID: \"26506959-bd2e-498f-9962-2ec4e27b4cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cnqgv"
Nov 03 08:43:50.529446 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:50.529412    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26506959-bd2e-498f-9962-2ec4e27b4cb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-cnqgv\" (UID: \"26506959-bd2e-498f-9962-2ec4e27b4cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cnqgv"
Nov 03 08:43:50.530022 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:50.530001    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26506959-bd2e-498f-9962-2ec4e27b4cb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-cnqgv\" (UID: \"26506959-bd2e-498f-9962-2ec4e27b4cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cnqgv"
Nov 03 08:43:50.531776 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:50.531739    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26506959-bd2e-498f-9962-2ec4e27b4cb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-cnqgv\" (UID: \"26506959-bd2e-498f-9962-2ec4e27b4cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cnqgv"
Nov 03 08:43:50.535791 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:50.535748    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8pjg\" (UniqueName: \"kubernetes.io/projected/26506959-bd2e-498f-9962-2ec4e27b4cb5-kube-api-access-z8pjg\") pod \"maas-api-revoke-sub-f03629db-cnqgv\" (UID: \"26506959-bd2e-498f-9962-2ec4e27b4cb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cnqgv"
Nov 03 08:43:50.654138 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:50.654113    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cnqgv"
Nov 03 08:43:50.926064 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:50.926040    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cnqgv"]
Nov 03 08:43:50.928278 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:43:50.928250    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26506959_bd2e_498f_9962_2ec4e27b4cb5.slice/crio-5fbdd34d1c79c30d572c6b42dde55f70766af8bbecf372262b1d58cd63b457cb WatchSource:0}: Error finding container 5fbdd34d1c79c30d572c6b42dde55f70766af8bbecf372262b1d58cd63b457cb: Status 404 returned error can't find the container with id 5fbdd34d1c79c30d572c6b42dde55f70766af8bbecf372262b1d58cd63b457cb
Nov 03 08:43:51.020236 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:51.020208    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cnqgv" event={"ID":"26506959-bd2e-498f-9962-2ec4e27b4cb5","Type":"ContainerStarted","Data":"5fbdd34d1c79c30d572c6b42dde55f70766af8bbecf372262b1d58cd63b457cb"}
Nov 03 08:43:52.026674 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:52.026642    5070 generic.go:356] "Generic (PLEG): container finished" podID="26506959-bd2e-498f-9962-2ec4e27b4cb5" containerID="046b69eed0f8e603a48f3a97e7f79a6d53602f61d2a15e7b3e7631ef75ce1d5a" exitCode=0
Nov 03 08:43:52.027043 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:52.026722    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cnqgv" event={"ID":"26506959-bd2e-498f-9962-2ec4e27b4cb5","Type":"ContainerDied","Data":"046b69eed0f8e603a48f3a97e7f79a6d53602f61d2a15e7b3e7631ef75ce1d5a"}
Nov 03 08:43:52.050747 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:43:52.050718    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:43:53.171091 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:53.171068    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cnqgv"
Nov 03 08:43:53.355122 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:53.355095    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26506959-bd2e-498f-9962-2ec4e27b4cb5-kube-api-access-z8pjg\" (UniqueName: \"kubernetes.io/projected/26506959-bd2e-498f-9962-2ec4e27b4cb5-kube-api-access-z8pjg\") pod \"26506959-bd2e-498f-9962-2ec4e27b4cb5\" (UID: \"26506959-bd2e-498f-9962-2ec4e27b4cb5\") "
Nov 03 08:43:53.355289 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:53.355164    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26506959-bd2e-498f-9962-2ec4e27b4cb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26506959-bd2e-498f-9962-2ec4e27b4cb5-maas-api-openshift-ca\") pod \"26506959-bd2e-498f-9962-2ec4e27b4cb5\" (UID: \"26506959-bd2e-498f-9962-2ec4e27b4cb5\") "
Nov 03 08:43:53.355289 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:53.355202    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26506959-bd2e-498f-9962-2ec4e27b4cb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26506959-bd2e-498f-9962-2ec4e27b4cb5-maas-api-xks-ca\") pod \"26506959-bd2e-498f-9962-2ec4e27b4cb5\" (UID: \"26506959-bd2e-498f-9962-2ec4e27b4cb5\") "
Nov 03 08:43:53.355616 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:53.355591    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26506959-bd2e-498f-9962-2ec4e27b4cb5-maas-api-openshift-ca" pod "26506959-bd2e-498f-9962-2ec4e27b4cb5" (UID: "26506959-bd2e-498f-9962-2ec4e27b4cb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:43:53.357197 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:53.357178    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26506959-bd2e-498f-9962-2ec4e27b4cb5-maas-api-xks-ca" pod "26506959-bd2e-498f-9962-2ec4e27b4cb5" (UID: "26506959-bd2e-498f-9962-2ec4e27b4cb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:43:53.357311 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:53.357240    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26506959-bd2e-498f-9962-2ec4e27b4cb5-kube-api-access-z8pjg" pod "26506959-bd2e-498f-9962-2ec4e27b4cb5" (UID: "26506959-bd2e-498f-9962-2ec4e27b4cb5"). InnerVolumeSpecName "kube-api-access-z8pjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:43:53.456739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:53.456715    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8pjg\" (UniqueName: \"kubernetes.io/projected/26506959-bd2e-498f-9962-2ec4e27b4cb5-kube-api-access-z8pjg\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:53.456739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:53.456736    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26506959-bd2e-498f-9962-2ec4e27b4cb5-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:53.456946 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:53.456747    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26506959-bd2e-498f-9962-2ec4e27b4cb5-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:43:54.037184 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:54.037162    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cnqgv"
Nov 03 08:43:54.037316 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:54.037210    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-cnqgv" event={"ID":"26506959-bd2e-498f-9962-2ec4e27b4cb5","Type":"ContainerDied","Data":"5fbdd34d1c79c30d572c6b42dde55f70766af8bbecf372262b1d58cd63b457cb"}
Nov 03 08:43:54.037316 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:43:54.037247    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fbdd34d1c79c30d572c6b42dde55f70766af8bbecf372262b1d58cd63b457cb"
Nov 03 08:44:01.269640 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:01.269606    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:44:05.253876 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:05.253839    5070 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-ln9hl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 08:44:05.254265 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:05.253888    5070 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-ln9hl" podUID="c4c16356-5579-4aac-aec8-c72808dbb1a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 08:44:06.218880 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:06.218849    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rw54d"]
Nov 03 08:44:06.219325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:06.219307    5070 cpu_manager.go:414] "containerMap: removing container" podUID="26506959-bd2e-498f-9962-2ec4e27b4cb5" containerName="revoke-keys"
Nov 03 08:44:06.219325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:06.219323    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26506959-bd2e-498f-9962-2ec4e27b4cb5" containerName="revoke-keys"
Nov 03 08:44:06.219507 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:06.219423    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="26506959-bd2e-498f-9962-2ec4e27b4cb5" containerName="revoke-keys"
Nov 03 08:44:06.222655 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:06.222631    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rw54d"
Nov 03 08:44:06.227284 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:06.227257    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:44:06.227840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:06.227791    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tp6bz\""
Nov 03 08:44:06.227840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:06.227797    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:44:06.227840 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:06.227827    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:44:06.231023 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:06.230996    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rw54d"]
Nov 03 08:44:06.295427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:06.294084    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027031ec-7182-4dc9-81b6-1823279c6ae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-rw54d\" (UID: \"027031ec-7182-4dc9-81b6-1823279c6ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rw54d"
Nov 03 08:44:06.295427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:06.294181    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027031ec-7182-4dc9-81b6-1823279c6ae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-rw54d\" (UID: \"027031ec-7182-4dc9-81b6-1823279c6ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rw54d"
Nov 03 08:44:06.295427 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:06.294305    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s65zv\" (UniqueName: \"kubernetes.io/projected/027031ec-7182-4dc9-81b6-1823279c6ae2-kube-api-access-s65zv\") pod \"maas-api-revoke-sub-0165d7d2-rw54d\" (UID: \"027031ec-7182-4dc9-81b6-1823279c6ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rw54d"
Nov 03 08:44:06.395615 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:06.395583    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027031ec-7182-4dc9-81b6-1823279c6ae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-rw54d\" (UID: \"027031ec-7182-4dc9-81b6-1823279c6ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rw54d"
Nov 03 08:44:06.395822 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:06.395661    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s65zv\" (UniqueName: \"kubernetes.io/projected/027031ec-7182-4dc9-81b6-1823279c6ae2-kube-api-access-s65zv\") pod \"maas-api-revoke-sub-0165d7d2-rw54d\" (UID: \"027031ec-7182-4dc9-81b6-1823279c6ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rw54d"
Nov 03 08:44:06.395822 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:06.395696    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027031ec-7182-4dc9-81b6-1823279c6ae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-rw54d\" (UID: \"027031ec-7182-4dc9-81b6-1823279c6ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rw54d"
Nov 03 08:44:06.396309 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:06.396288    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027031ec-7182-4dc9-81b6-1823279c6ae2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-rw54d\" (UID: \"027031ec-7182-4dc9-81b6-1823279c6ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rw54d"
Nov 03 08:44:06.398033 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:06.398009    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027031ec-7182-4dc9-81b6-1823279c6ae2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-rw54d\" (UID: \"027031ec-7182-4dc9-81b6-1823279c6ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rw54d"
Nov 03 08:44:06.403661 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:06.403638    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s65zv\" (UniqueName: \"kubernetes.io/projected/027031ec-7182-4dc9-81b6-1823279c6ae2-kube-api-access-s65zv\") pod \"maas-api-revoke-sub-0165d7d2-rw54d\" (UID: \"027031ec-7182-4dc9-81b6-1823279c6ae2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rw54d"
Nov 03 08:44:06.540868 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:06.540788    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rw54d"
Nov 03 08:44:06.815018 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:06.814993    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rw54d"]
Nov 03 08:44:06.817113 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:44:06.817085    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027031ec_7182_4dc9_81b6_1823279c6ae2.slice/crio-c79cfc1e5599b7afab509b560dd7b9a71388a91949a6e499b57fca0f2c8c2d08 WatchSource:0}: Error finding container c79cfc1e5599b7afab509b560dd7b9a71388a91949a6e499b57fca0f2c8c2d08: Status 404 returned error can't find the container with id c79cfc1e5599b7afab509b560dd7b9a71388a91949a6e499b57fca0f2c8c2d08
Nov 03 08:44:07.029024 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:07.028998    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq"]
Nov 03 08:44:07.031156 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:07.031135    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad114a5-mv5vq"]
Nov 03 08:44:07.137130 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:07.137105    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rw54d" event={"ID":"027031ec-7182-4dc9-81b6-1823279c6ae2","Type":"ContainerStarted","Data":"7f8eb5d131f3f5df9c7aa3fbae362c6f85112db6ad0ad98fb6ecdbd847fee5b7"}
Nov 03 08:44:07.137226 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:07.137148    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rw54d" event={"ID":"027031ec-7182-4dc9-81b6-1823279c6ae2","Type":"ContainerStarted","Data":"c79cfc1e5599b7afab509b560dd7b9a71388a91949a6e499b57fca0f2c8c2d08"}
Nov 03 08:44:07.151465 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:07.151398    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rw54d" podStartSLOduration=1.151382763 podStartE2EDuration="1.151382763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:44:07.150435191 +0000 UTC m=+3045.264899123" watchObservedRunningTime="2026-11-03 08:44:07.151382763 +0000 UTC m=+3045.265846694"
Nov 03 08:44:08.024037 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:08.023999    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da7af76a-f748-4e11-b6b6-2555511651cc" path="/var/lib/kubelet/pods/da7af76a-f748-4e11-b6b6-2555511651cc/volumes"
Nov 03 08:44:08.144562 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:08.144534    5070 generic.go:356] "Generic (PLEG): container finished" podID="027031ec-7182-4dc9-81b6-1823279c6ae2" containerID="7f8eb5d131f3f5df9c7aa3fbae362c6f85112db6ad0ad98fb6ecdbd847fee5b7" exitCode=0
Nov 03 08:44:08.144714 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:08.144599    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rw54d" event={"ID":"027031ec-7182-4dc9-81b6-1823279c6ae2","Type":"ContainerDied","Data":"7f8eb5d131f3f5df9c7aa3fbae362c6f85112db6ad0ad98fb6ecdbd847fee5b7"}
Nov 03 08:44:09.311089 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:09.311063    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rw54d"
Nov 03 08:44:09.425246 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:09.425187    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/027031ec-7182-4dc9-81b6-1823279c6ae2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027031ec-7182-4dc9-81b6-1823279c6ae2-maas-api-openshift-ca\") pod \"027031ec-7182-4dc9-81b6-1823279c6ae2\" (UID: \"027031ec-7182-4dc9-81b6-1823279c6ae2\") "
Nov 03 08:44:09.425246 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:09.425234    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/027031ec-7182-4dc9-81b6-1823279c6ae2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027031ec-7182-4dc9-81b6-1823279c6ae2-maas-api-xks-ca\") pod \"027031ec-7182-4dc9-81b6-1823279c6ae2\" (UID: \"027031ec-7182-4dc9-81b6-1823279c6ae2\") "
Nov 03 08:44:09.425403 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:09.425275    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027031ec-7182-4dc9-81b6-1823279c6ae2-kube-api-access-s65zv\" (UniqueName: \"kubernetes.io/projected/027031ec-7182-4dc9-81b6-1823279c6ae2-kube-api-access-s65zv\") pod \"027031ec-7182-4dc9-81b6-1823279c6ae2\" (UID: \"027031ec-7182-4dc9-81b6-1823279c6ae2\") "
Nov 03 08:44:09.425597 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:09.425572    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027031ec-7182-4dc9-81b6-1823279c6ae2-maas-api-openshift-ca" pod "027031ec-7182-4dc9-81b6-1823279c6ae2" (UID: "027031ec-7182-4dc9-81b6-1823279c6ae2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:44:09.427346 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:09.427319    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027031ec-7182-4dc9-81b6-1823279c6ae2-kube-api-access-s65zv" pod "027031ec-7182-4dc9-81b6-1823279c6ae2" (UID: "027031ec-7182-4dc9-81b6-1823279c6ae2"). InnerVolumeSpecName "kube-api-access-s65zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:44:09.427346 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:09.427321    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027031ec-7182-4dc9-81b6-1823279c6ae2-maas-api-xks-ca" pod "027031ec-7182-4dc9-81b6-1823279c6ae2" (UID: "027031ec-7182-4dc9-81b6-1823279c6ae2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:44:09.526270 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:09.526242    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027031ec-7182-4dc9-81b6-1823279c6ae2-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:44:09.526270 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:09.526267    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s65zv\" (UniqueName: \"kubernetes.io/projected/027031ec-7182-4dc9-81b6-1823279c6ae2-kube-api-access-s65zv\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:44:09.526418 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:09.526283    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027031ec-7182-4dc9-81b6-1823279c6ae2-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:44:10.155714 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:10.155683    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rw54d"
Nov 03 08:44:10.155881 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:10.155737    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rw54d" event={"ID":"027031ec-7182-4dc9-81b6-1823279c6ae2","Type":"ContainerDied","Data":"c79cfc1e5599b7afab509b560dd7b9a71388a91949a6e499b57fca0f2c8c2d08"}
Nov 03 08:44:10.155881 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:10.155798    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c79cfc1e5599b7afab509b560dd7b9a71388a91949a6e499b57fca0f2c8c2d08"
Nov 03 08:44:16.874893 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:16.874860    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j9gz7"]
Nov 03 08:44:16.875305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:16.875275    5070 cpu_manager.go:414] "containerMap: removing container" podUID="027031ec-7182-4dc9-81b6-1823279c6ae2" containerName="revoke-keys"
Nov 03 08:44:16.875305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:16.875285    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027031ec-7182-4dc9-81b6-1823279c6ae2" containerName="revoke-keys"
Nov 03 08:44:16.875403 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:16.875382    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="027031ec-7182-4dc9-81b6-1823279c6ae2" containerName="revoke-keys"
Nov 03 08:44:16.878770 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:16.878744    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j9gz7"
Nov 03 08:44:16.882405 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:16.882375    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:44:16.882674 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:16.882656    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 08:44:16.882820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:16.882668    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:44:16.882820 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:16.882736    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tp6bz\""
Nov 03 08:44:16.886121 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:16.886091    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j9gz7"]
Nov 03 08:44:16.987426 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:16.987395    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed78ff2-1e9e-4576-9fd9-f05ea1892b5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-j9gz7\" (UID: \"bed78ff2-1e9e-4576-9fd9-f05ea1892b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j9gz7"
Nov 03 08:44:16.987596 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:16.987445    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w7lp\" (UniqueName: \"kubernetes.io/projected/bed78ff2-1e9e-4576-9fd9-f05ea1892b5b-kube-api-access-9w7lp\") pod \"maas-api-revoke-sub-6b19338f-j9gz7\" (UID: \"bed78ff2-1e9e-4576-9fd9-f05ea1892b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j9gz7"
Nov 03 08:44:16.987596 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:16.987555    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed78ff2-1e9e-4576-9fd9-f05ea1892b5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-j9gz7\" (UID: \"bed78ff2-1e9e-4576-9fd9-f05ea1892b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j9gz7"
Nov 03 08:44:17.088163 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:17.088130    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w7lp\" (UniqueName: \"kubernetes.io/projected/bed78ff2-1e9e-4576-9fd9-f05ea1892b5b-kube-api-access-9w7lp\") pod \"maas-api-revoke-sub-6b19338f-j9gz7\" (UID: \"bed78ff2-1e9e-4576-9fd9-f05ea1892b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j9gz7"
Nov 03 08:44:17.088325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:17.088187    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed78ff2-1e9e-4576-9fd9-f05ea1892b5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-j9gz7\" (UID: \"bed78ff2-1e9e-4576-9fd9-f05ea1892b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j9gz7"
Nov 03 08:44:17.088325 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:17.088285    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed78ff2-1e9e-4576-9fd9-f05ea1892b5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-j9gz7\" (UID: \"bed78ff2-1e9e-4576-9fd9-f05ea1892b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j9gz7"
Nov 03 08:44:17.088861 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:17.088834    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed78ff2-1e9e-4576-9fd9-f05ea1892b5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-j9gz7\" (UID: \"bed78ff2-1e9e-4576-9fd9-f05ea1892b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j9gz7"
Nov 03 08:44:17.090597 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:17.090577    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed78ff2-1e9e-4576-9fd9-f05ea1892b5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-j9gz7\" (UID: \"bed78ff2-1e9e-4576-9fd9-f05ea1892b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j9gz7"
Nov 03 08:44:17.095463 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:17.095441    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w7lp\" (UniqueName: \"kubernetes.io/projected/bed78ff2-1e9e-4576-9fd9-f05ea1892b5b-kube-api-access-9w7lp\") pod \"maas-api-revoke-sub-6b19338f-j9gz7\" (UID: \"bed78ff2-1e9e-4576-9fd9-f05ea1892b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j9gz7"
Nov 03 08:44:17.194842 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:17.194785    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j9gz7"
Nov 03 08:44:17.469356 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:17.469331    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j9gz7"]
Nov 03 08:44:17.471746 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:44:17.471714    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed78ff2_1e9e_4576_9fd9_f05ea1892b5b.slice/crio-d9f3d88320fe3760b255af4f477234148fbca811e0c11c44448441f86957c7bc WatchSource:0}: Error finding container d9f3d88320fe3760b255af4f477234148fbca811e0c11c44448441f86957c7bc: Status 404 returned error can't find the container with id d9f3d88320fe3760b255af4f477234148fbca811e0c11c44448441f86957c7bc
Nov 03 08:44:18.202774 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:18.202714    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j9gz7" event={"ID":"bed78ff2-1e9e-4576-9fd9-f05ea1892b5b","Type":"ContainerStarted","Data":"b95db11baf038f702b37efef00176db3c61a13bfbe97aae88cf4469cb2ff2235"}
Nov 03 08:44:18.203143 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:18.202787    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j9gz7" event={"ID":"bed78ff2-1e9e-4576-9fd9-f05ea1892b5b","Type":"ContainerStarted","Data":"d9f3d88320fe3760b255af4f477234148fbca811e0c11c44448441f86957c7bc"}
Nov 03 08:44:19.209270 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:19.209234    5070 generic.go:356] "Generic (PLEG): container finished" podID="bed78ff2-1e9e-4576-9fd9-f05ea1892b5b" containerID="b95db11baf038f702b37efef00176db3c61a13bfbe97aae88cf4469cb2ff2235" exitCode=0
Nov 03 08:44:19.209693 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:19.209309    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j9gz7" event={"ID":"bed78ff2-1e9e-4576-9fd9-f05ea1892b5b","Type":"ContainerDied","Data":"b95db11baf038f702b37efef00176db3c61a13bfbe97aae88cf4469cb2ff2235"}
Nov 03 08:44:20.358929 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:20.358905    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j9gz7"
Nov 03 08:44:20.414007 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:20.413979    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bed78ff2-1e9e-4576-9fd9-f05ea1892b5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed78ff2-1e9e-4576-9fd9-f05ea1892b5b-maas-api-xks-ca\") pod \"bed78ff2-1e9e-4576-9fd9-f05ea1892b5b\" (UID: \"bed78ff2-1e9e-4576-9fd9-f05ea1892b5b\") "
Nov 03 08:44:20.414145 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:20.414071    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bed78ff2-1e9e-4576-9fd9-f05ea1892b5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed78ff2-1e9e-4576-9fd9-f05ea1892b5b-maas-api-openshift-ca\") pod \"bed78ff2-1e9e-4576-9fd9-f05ea1892b5b\" (UID: \"bed78ff2-1e9e-4576-9fd9-f05ea1892b5b\") "
Nov 03 08:44:20.414145 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:20.414122    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bed78ff2-1e9e-4576-9fd9-f05ea1892b5b-kube-api-access-9w7lp\" (UniqueName: \"kubernetes.io/projected/bed78ff2-1e9e-4576-9fd9-f05ea1892b5b-kube-api-access-9w7lp\") pod \"bed78ff2-1e9e-4576-9fd9-f05ea1892b5b\" (UID: \"bed78ff2-1e9e-4576-9fd9-f05ea1892b5b\") "
Nov 03 08:44:20.414405 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:20.414380    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed78ff2-1e9e-4576-9fd9-f05ea1892b5b-maas-api-openshift-ca" pod "bed78ff2-1e9e-4576-9fd9-f05ea1892b5b" (UID: "bed78ff2-1e9e-4576-9fd9-f05ea1892b5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:44:20.414523 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:20.414509    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed78ff2-1e9e-4576-9fd9-f05ea1892b5b-maas-api-openshift-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:44:20.416548 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:20.416522    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed78ff2-1e9e-4576-9fd9-f05ea1892b5b-maas-api-xks-ca" pod "bed78ff2-1e9e-4576-9fd9-f05ea1892b5b" (UID: "bed78ff2-1e9e-4576-9fd9-f05ea1892b5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 08:44:20.416548 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:20.416540    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed78ff2-1e9e-4576-9fd9-f05ea1892b5b-kube-api-access-9w7lp" pod "bed78ff2-1e9e-4576-9fd9-f05ea1892b5b" (UID: "bed78ff2-1e9e-4576-9fd9-f05ea1892b5b"). InnerVolumeSpecName "kube-api-access-9w7lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:44:20.515294 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:20.515244    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed78ff2-1e9e-4576-9fd9-f05ea1892b5b-maas-api-xks-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:44:20.515294 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:20.515266    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w7lp\" (UniqueName: \"kubernetes.io/projected/bed78ff2-1e9e-4576-9fd9-f05ea1892b5b-kube-api-access-9w7lp\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:44:21.222046 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:21.222013    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j9gz7" event={"ID":"bed78ff2-1e9e-4576-9fd9-f05ea1892b5b","Type":"ContainerDied","Data":"d9f3d88320fe3760b255af4f477234148fbca811e0c11c44448441f86957c7bc"}
Nov 03 08:44:21.222230 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:21.222065    5070 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9f3d88320fe3760b255af4f477234148fbca811e0c11c44448441f86957c7bc"
Nov 03 08:44:21.222230 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:21.222026    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j9gz7"
Nov 03 08:44:22.050669 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:44:22.050637    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:44:27.776855 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:27.776818    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-qp9pc"]
Nov 03 08:44:28.160338 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:28.160304    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6995574766-s7dx9_b148fccf-b3ae-4b52-9df1-21e85f7e3fa3/authorino/0.log"
Nov 03 08:44:39.115830 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:39.115794    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-mdwfw_0d26093c-a60a-4cd8-9d43-8ecded62d895/server/1.log"
Nov 03 08:44:39.845488 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:39.845455    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-mdwfw_0d26093c-a60a-4cd8-9d43-8ecded62d895/server/0.log"
Nov 03 08:44:40.741739 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:40.741706    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-rw54d_027031ec-7182-4dc9-81b6-1823279c6ae2/revoke-keys/0.log"
Nov 03 08:44:40.905469 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:40.905444    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-94lwl_88ab565e-cc66-4c3e-a7b0-27ed4880d00d/revoke-keys/0.log"
Nov 03 08:44:41.061348 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:41.061267    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-55n56_2b0dd530-62a7-457e-b769-a677c51e79d7/revoke-keys/0.log"
Nov 03 08:44:41.223262 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:41.223239    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-j9gz7_bed78ff2-1e9e-4576-9fd9-f05ea1892b5b/revoke-keys/0.log"
Nov 03 08:44:41.552817 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:41.552793    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-cnqgv_26506959-bd2e-498f-9962-2ec4e27b4cb5/revoke-keys/0.log"
Nov 03 08:44:43.163846 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:43.163798    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6995574766-s7dx9_b148fccf-b3ae-4b52-9df1-21e85f7e3fa3/authorino/0.log"
Nov 03 08:44:43.391008 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:43.390978    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-6kbrb_17267346-b1b3-475b-9d89-1fda6e36aab5/manager/1.log"
Nov 03 08:44:44.142594 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:44.142560    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-qp9pc_7d888577-bc72-4581-a858-b43bd5d7290c/limitador/0.log"
Nov 03 08:44:44.728610 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:44.728575    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-6kbrb_17267346-b1b3-475b-9d89-1fda6e36aab5/manager/0.log"
Nov 03 08:44:45.779424 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:45.779396    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cf769c4c7-l2kwc_0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f/kube-auth-proxy/0.log"
Nov 03 08:44:46.112509 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:46.112479    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-87snj_0c9ad9ff-3d0f-4994-829c-81fe102cc3de/istio-proxy/0.log"
Nov 03 08:44:46.283775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:46.283734    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-8rcwm_a86a6470-9786-4e2d-8668-62ce0dbbb596/istio-proxy/0.log"
Nov 03 08:44:47.007775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:47.007726    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-ln9hl_c4c16356-5579-4aac-aec8-c72808dbb1a2/istio-proxy/0.log"
Nov 03 08:44:48.202165 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:48.202130    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79698c9d55-m2lbh_ce890601-127d-4995-9487-635e7aab04f8/router/0.log"
Nov 03 08:44:49.097234 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:49.097194    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4_d50d86f3-08a1-419c-a77b-d7223e0567c7/main/0.log"
Nov 03 08:44:49.441132 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:49.441102    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-gstjn_6960b432-0c5c-4c03-bcc5-dec9ff9466c7/main/0.log"
Nov 03 08:44:49.781161 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:49.781093    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz_33f324bd-79b1-4540-a090-7e4a9d92f10b/main/0.log"
Nov 03 08:44:49.966368 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:49.966332    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-qfg9j_67eb2350-8895-4e7d-b1f2-e78e1ff5cf23/main/0.log"
Nov 03 08:44:50.326041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:50.326010    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t_66d4513c-da3f-430c-91e3-bb8d782d5917/main/0.log"
Nov 03 08:44:51.167522 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:44:51.167485    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6995574766-s7dx9_b148fccf-b3ae-4b52-9df1-21e85f7e3fa3/authorino/0.log"
Nov 03 08:44:52.050520 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:44:52.050483    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:45:00.149433 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:00.149397    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g"]
Nov 03 08:45:00.150098 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:00.150077    5070 cpu_manager.go:414] "containerMap: removing container" podUID="bed78ff2-1e9e-4576-9fd9-f05ea1892b5b" containerName="revoke-keys"
Nov 03 08:45:00.150163 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:00.150103    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed78ff2-1e9e-4576-9fd9-f05ea1892b5b" containerName="revoke-keys"
Nov 03 08:45:00.150246 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:00.150231    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed78ff2-1e9e-4576-9fd9-f05ea1892b5b" containerName="revoke-keys"
Nov 03 08:45:00.155207 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:00.155181    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g"
Nov 03 08:45:00.158241 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:00.158216    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 08:45:00.158400 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:00.158290    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 08:45:00.158498 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:00.158479    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tp6bz\""
Nov 03 08:45:00.161691 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:00.161668    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g"]
Nov 03 08:45:00.286596 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:00.286563    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghltw\" (UniqueName: \"kubernetes.io/projected/f5bf12cb-42ae-44d1-9bbc-dea3c634690a-kube-api-access-ghltw\") pod \"maas-api-key-cleanup-29894925-zvb5g\" (UID: \"f5bf12cb-42ae-44d1-9bbc-dea3c634690a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g"
Nov 03 08:45:00.286789 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:00.286620    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5bf12cb-42ae-44d1-9bbc-dea3c634690a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894925-zvb5g\" (UID: \"f5bf12cb-42ae-44d1-9bbc-dea3c634690a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g"
Nov 03 08:45:00.388118 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:00.388084    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghltw\" (UniqueName: \"kubernetes.io/projected/f5bf12cb-42ae-44d1-9bbc-dea3c634690a-kube-api-access-ghltw\") pod \"maas-api-key-cleanup-29894925-zvb5g\" (UID: \"f5bf12cb-42ae-44d1-9bbc-dea3c634690a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g"
Nov 03 08:45:00.388278 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:00.388173    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5bf12cb-42ae-44d1-9bbc-dea3c634690a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894925-zvb5g\" (UID: \"f5bf12cb-42ae-44d1-9bbc-dea3c634690a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g"
Nov 03 08:45:00.388796 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:00.388750    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5bf12cb-42ae-44d1-9bbc-dea3c634690a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894925-zvb5g\" (UID: \"f5bf12cb-42ae-44d1-9bbc-dea3c634690a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g"
Nov 03 08:45:00.395836 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:00.395820    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghltw\" (UniqueName: \"kubernetes.io/projected/f5bf12cb-42ae-44d1-9bbc-dea3c634690a-kube-api-access-ghltw\") pod \"maas-api-key-cleanup-29894925-zvb5g\" (UID: \"f5bf12cb-42ae-44d1-9bbc-dea3c634690a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g"
Nov 03 08:45:00.470514 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:00.470464    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g"
Nov 03 08:45:00.609389 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:00.609366    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-mdwfw_0d26093c-a60a-4cd8-9d43-8ecded62d895/server/1.log"
Nov 03 08:45:00.743937 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:00.743909    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g"]
Nov 03 08:45:00.745733 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:45:00.745708    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bf12cb_42ae_44d1_9bbc_dea3c634690a.slice/crio-76c51c9a213bf5e8764d0539081b3c9b50d2af78f00539bb05cf7d816027973f WatchSource:0}: Error finding container 76c51c9a213bf5e8764d0539081b3c9b50d2af78f00539bb05cf7d816027973f: Status 404 returned error can't find the container with id 76c51c9a213bf5e8764d0539081b3c9b50d2af78f00539bb05cf7d816027973f
Nov 03 08:45:01.477415 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:01.477386    5070 generic.go:356] "Generic (PLEG): container finished" podID="f5bf12cb-42ae-44d1-9bbc-dea3c634690a" containerID="1e7c614a02377f96a43cc11fc2b94820dfb58f7601efcf0d83c21bdcae18585e" exitCode=60
Nov 03 08:45:01.477701 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:01.477467    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g" event={"ID":"f5bf12cb-42ae-44d1-9bbc-dea3c634690a","Type":"ContainerDied","Data":"1e7c614a02377f96a43cc11fc2b94820dfb58f7601efcf0d83c21bdcae18585e"}
Nov 03 08:45:01.477701 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:01.477533    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g" event={"ID":"f5bf12cb-42ae-44d1-9bbc-dea3c634690a","Type":"ContainerStarted","Data":"76c51c9a213bf5e8764d0539081b3c9b50d2af78f00539bb05cf7d816027973f"}
Nov 03 08:45:01.477701 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:01.477656    5070 scope.go:122] "RemoveContainer" containerID="1e7c614a02377f96a43cc11fc2b94820dfb58f7601efcf0d83c21bdcae18585e"
Nov 03 08:45:01.831305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:01.831276    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29894925-zvb5g_f5bf12cb-42ae-44d1-9bbc-dea3c634690a/cleanup/0.log"
Nov 03 08:45:01.998285 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:01.998260    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-rw54d_027031ec-7182-4dc9-81b6-1823279c6ae2/revoke-keys/0.log"
Nov 03 08:45:02.170262 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:02.170239    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-94lwl_88ab565e-cc66-4c3e-a7b0-27ed4880d00d/revoke-keys/0.log"
Nov 03 08:45:02.334296 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:02.334272    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-55n56_2b0dd530-62a7-457e-b769-a677c51e79d7/revoke-keys/0.log"
Nov 03 08:45:02.485283 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:02.485204    5070 generic.go:356] "Generic (PLEG): container finished" podID="f5bf12cb-42ae-44d1-9bbc-dea3c634690a" containerID="9a2e8d5cd291c879c775623e6e4822b797f831670ed9fac70d4f4a8f1c653e09" exitCode=60
Nov 03 08:45:02.485728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:02.485288    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g" event={"ID":"f5bf12cb-42ae-44d1-9bbc-dea3c634690a","Type":"ContainerDied","Data":"9a2e8d5cd291c879c775623e6e4822b797f831670ed9fac70d4f4a8f1c653e09"}
Nov 03 08:45:02.485728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:02.485357    5070 scope.go:122] "RemoveContainer" containerID="1e7c614a02377f96a43cc11fc2b94820dfb58f7601efcf0d83c21bdcae18585e"
Nov 03 08:45:02.485728 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:02.485487    5070 scope.go:122] "RemoveContainer" containerID="9a2e8d5cd291c879c775623e6e4822b797f831670ed9fac70d4f4a8f1c653e09"
Nov 03 08:45:02.485934 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:45:02.485751    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894925-zvb5g_odh-ai-gateway-infra(f5bf12cb-42ae-44d1-9bbc-dea3c634690a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g" podUID="f5bf12cb-42ae-44d1-9bbc-dea3c634690a"
Nov 03 08:45:02.502591 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:02.502571    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-j9gz7_bed78ff2-1e9e-4576-9fd9-f05ea1892b5b/revoke-keys/0.log"
Nov 03 08:45:02.844818 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:02.844723    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-cnqgv_26506959-bd2e-498f-9962-2ec4e27b4cb5/revoke-keys/0.log"
Nov 03 08:45:03.493291 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:03.493267    5070 scope.go:122] "RemoveContainer" containerID="9a2e8d5cd291c879c775623e6e4822b797f831670ed9fac70d4f4a8f1c653e09"
Nov 03 08:45:03.493671 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:45:03.493443    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894925-zvb5g_odh-ai-gateway-infra(f5bf12cb-42ae-44d1-9bbc-dea3c634690a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g" podUID="f5bf12cb-42ae-44d1-9bbc-dea3c634690a"
Nov 03 08:45:04.072272 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:04.072241    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6995574766-s7dx9_b148fccf-b3ae-4b52-9df1-21e85f7e3fa3/authorino/0.log"
Nov 03 08:45:04.280009 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:04.279982    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-6kbrb_17267346-b1b3-475b-9d89-1fda6e36aab5/manager/1.log"
Nov 03 08:45:04.968308 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:04.968275    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-qp9pc_7d888577-bc72-4581-a858-b43bd5d7290c/limitador/0.log"
Nov 03 08:45:06.190060 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:06.190028    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cf769c4c7-l2kwc_0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f/kube-auth-proxy/0.log"
Nov 03 08:45:06.538790 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:06.538667    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-87snj_0c9ad9ff-3d0f-4994-829c-81fe102cc3de/istio-proxy/0.log"
Nov 03 08:45:06.716588 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:06.716556    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-8rcwm_a86a6470-9786-4e2d-8668-62ce0dbbb596/istio-proxy/0.log"
Nov 03 08:45:07.390729 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:07.390701    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-ln9hl_c4c16356-5579-4aac-aec8-c72808dbb1a2/istio-proxy/0.log"
Nov 03 08:45:08.603713 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:08.603674    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79698c9d55-m2lbh_ce890601-127d-4995-9487-635e7aab04f8/router/0.log"
Nov 03 08:45:09.119305 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:09.119265    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-qzdb4_d50d86f3-08a1-419c-a77b-d7223e0567c7/main/0.log"
Nov 03 08:45:09.453822 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:09.453729    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-gstjn_6960b432-0c5c-4c03-bcc5-dec9ff9466c7/main/0.log"
Nov 03 08:45:09.779848 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:09.779749    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8m6chz_33f324bd-79b1-4540-a090-7e4a9d92f10b/main/0.log"
Nov 03 08:45:09.929822 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:09.929794    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-qfg9j_67eb2350-8895-4e7d-b1f2-e78e1ff5cf23/main/0.log"
Nov 03 08:45:10.285300 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:10.285270    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-xmf5t_66d4513c-da3f-430c-91e3-bb8d782d5917/main/0.log"
Nov 03 08:45:12.024512 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:12.024480    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56"]
Nov 03 08:45:12.024980 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:12.024873    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-55n56"]
Nov 03 08:45:14.024689 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:14.024650    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b0dd530-62a7-457e-b769-a677c51e79d7" path="/var/lib/kubelet/pods/2b0dd530-62a7-457e-b769-a677c51e79d7/volumes"
Nov 03 08:45:15.535145 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:15.535118    5070 scope.go:122] "RemoveContainer" containerID="855f1115690adbd8b54e74d0f3f2cb77e9ac51dff19eb3f92c0ba4db8f385e9a"
Nov 03 08:45:15.543947 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:15.543927    5070 scope.go:122] "RemoveContainer" containerID="da17196c8656cd1679c7b191c3b30911ef3d0938254efee389a1c428fc4cc60b"
Nov 03 08:45:16.017877 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:16.017851    5070 scope.go:122] "RemoveContainer" containerID="9a2e8d5cd291c879c775623e6e4822b797f831670ed9fac70d4f4a8f1c653e09"
Nov 03 08:45:16.568140 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:16.568053    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g" event={"ID":"f5bf12cb-42ae-44d1-9bbc-dea3c634690a","Type":"ContainerStarted","Data":"97027cea01f6bccd132c966a11386cd52289645644de14e3acd208159c7aa21a"}
Nov 03 08:45:16.585250 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:16.585196    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g" podStartSLOduration=16.585182776 podStartE2EDuration="16.585182776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 08:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:45:16.582988878 +0000 UTC m=+3114.697452851" watchObservedRunningTime="2026-11-03 08:45:16.585182776 +0000 UTC m=+3114.699646721"
Nov 03 08:45:17.576260 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:17.576224    5070 generic.go:356] "Generic (PLEG): container finished" podID="f5bf12cb-42ae-44d1-9bbc-dea3c634690a" containerID="97027cea01f6bccd132c966a11386cd52289645644de14e3acd208159c7aa21a" exitCode=60
Nov 03 08:45:17.576710 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:17.576298    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g" event={"ID":"f5bf12cb-42ae-44d1-9bbc-dea3c634690a","Type":"ContainerDied","Data":"97027cea01f6bccd132c966a11386cd52289645644de14e3acd208159c7aa21a"}
Nov 03 08:45:17.576710 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:17.576362    5070 scope.go:122] "RemoveContainer" containerID="9a2e8d5cd291c879c775623e6e4822b797f831670ed9fac70d4f4a8f1c653e09"
Nov 03 08:45:17.576710 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:17.576631    5070 scope.go:122] "RemoveContainer" containerID="97027cea01f6bccd132c966a11386cd52289645644de14e3acd208159c7aa21a"
Nov 03 08:45:17.576940 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:45:17.576858    5070 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894925-zvb5g_odh-ai-gateway-infra(f5bf12cb-42ae-44d1-9bbc-dea3c634690a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g" podUID="f5bf12cb-42ae-44d1-9bbc-dea3c634690a"
Nov 03 08:45:17.593585 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:17.593563    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g"]
Nov 03 08:45:18.730988 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:18.730966    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g"
Nov 03 08:45:18.855908 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:18.855876    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5bf12cb-42ae-44d1-9bbc-dea3c634690a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5bf12cb-42ae-44d1-9bbc-dea3c634690a-maas-api-service-ca\") pod \"f5bf12cb-42ae-44d1-9bbc-dea3c634690a\" (UID: \"f5bf12cb-42ae-44d1-9bbc-dea3c634690a\") "
Nov 03 08:45:18.856076 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:18.855995    5070 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5bf12cb-42ae-44d1-9bbc-dea3c634690a-kube-api-access-ghltw\" (UniqueName: \"kubernetes.io/projected/f5bf12cb-42ae-44d1-9bbc-dea3c634690a-kube-api-access-ghltw\") pod \"f5bf12cb-42ae-44d1-9bbc-dea3c634690a\" (UID: \"f5bf12cb-42ae-44d1-9bbc-dea3c634690a\") "
Nov 03 08:45:18.856293 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:18.856270    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5bf12cb-42ae-44d1-9bbc-dea3c634690a-maas-api-service-ca" pod "f5bf12cb-42ae-44d1-9bbc-dea3c634690a" (UID: "f5bf12cb-42ae-44d1-9bbc-dea3c634690a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 08:45:18.856373 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:18.856360    5070 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5bf12cb-42ae-44d1-9bbc-dea3c634690a-maas-api-service-ca\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:45:18.857994 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:18.857968    5070 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5bf12cb-42ae-44d1-9bbc-dea3c634690a-kube-api-access-ghltw" pod "f5bf12cb-42ae-44d1-9bbc-dea3c634690a" (UID: "f5bf12cb-42ae-44d1-9bbc-dea3c634690a"). InnerVolumeSpecName "kube-api-access-ghltw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 08:45:18.957527 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:18.957500    5070 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghltw\" (UniqueName: \"kubernetes.io/projected/f5bf12cb-42ae-44d1-9bbc-dea3c634690a-kube-api-access-ghltw\") on node \"ip-10-0-140-123.us-east-2.compute.internal\" DevicePath \"\""
Nov 03 08:45:19.594189 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:19.594157    5070 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g"
Nov 03 08:45:19.594189 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:19.594171    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g" event={"ID":"f5bf12cb-42ae-44d1-9bbc-dea3c634690a","Type":"ContainerDied","Data":"76c51c9a213bf5e8764d0539081b3c9b50d2af78f00539bb05cf7d816027973f"}
Nov 03 08:45:19.594409 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:19.594261    5070 scope.go:122] "RemoveContainer" containerID="97027cea01f6bccd132c966a11386cd52289645644de14e3acd208159c7aa21a"
Nov 03 08:45:19.617533 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:19.617506    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g"]
Nov 03 08:45:19.623511 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:19.623487    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894925-zvb5g"]
Nov 03 08:45:20.027964 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:20.025463    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5bf12cb-42ae-44d1-9bbc-dea3c634690a" path="/var/lib/kubelet/pods/f5bf12cb-42ae-44d1-9bbc-dea3c634690a/volumes"
Nov 03 08:45:22.050554 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:45:22.050521    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:45:26.024188 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:26.024157    5070 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl"]
Nov 03 08:45:26.024573 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:26.024200    5070 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-94lwl"]
Nov 03 08:45:28.024142 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:45:28.024109    5070 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88ab565e-cc66-4c3e-a7b0-27ed4880d00d" path="/var/lib/kubelet/pods/88ab565e-cc66-4c3e-a7b0-27ed4880d00d/volumes"
Nov 03 08:45:52.050190 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:45:52.050151    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:46:15.568694 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:15.568664    5070 scope.go:122] "RemoveContainer" containerID="cd5f97c2c9cca4acbade74941067be23e1a7764d8599a4f1220f11f9e038f386"
Nov 03 08:46:22.052211 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:46:22.052177    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:46:29.946966 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:29.946935    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ljbh2_dcdc19b6-bba8-4d45-bbe0-0ea2ebbee9df/global-pull-secret-syncer/0.log"
Nov 03 08:46:29.980694 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:29.980673    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-kr2q8_edf55612-a9f0-4fd5-ab00-ac2271acfb30/connection-checker/0.log"
Nov 03 08:46:30.005098 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:30.005075    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-nwtkn_08817fd9-0879-4b77-a767-5f4c4ecf8d5b/connection-checker/0.log"
Nov 03 08:46:30.028115 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:30.028092    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-v2z4k_20b9100c-2539-4227-8f2a-c93660540664/connection-checker/0.log"
Nov 03 08:46:30.150316 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:30.150291    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9j2f9_676416b5-105e-4650-baed-92baf169a9a6/konnectivity-agent/0.log"
Nov 03 08:46:30.224422 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:30.224361    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-123.us-east-2.compute.internal_fc5342d3d37d4243b5710bd6f9020da4/haproxy/0.log"
Nov 03 08:46:37.354924 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:37.354848    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6995574766-s7dx9_b148fccf-b3ae-4b52-9df1-21e85f7e3fa3/authorino/0.log"
Nov 03 08:46:37.383499 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:37.383470    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-6kbrb_17267346-b1b3-475b-9d89-1fda6e36aab5/manager/1.log"
Nov 03 08:46:37.389544 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:37.389514    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-6kbrb_17267346-b1b3-475b-9d89-1fda6e36aab5/manager/0.log"
Nov 03 08:46:37.556506 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:37.556486    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-qp9pc_7d888577-bc72-4581-a858-b43bd5d7290c/limitador/0.log"
Nov 03 08:46:38.995334 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:38.995292    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-f68ks_c383d3eb-10c7-4fe5-99b6-04c64472ef0a/cluster-monitoring-operator/0.log"
Nov 03 08:46:39.163369 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.163331    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4sbrg_60772c10-1acb-489a-89bc-14388a92c001/node-exporter/0.log"
Nov 03 08:46:39.184775 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.184710    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4sbrg_60772c10-1acb-489a-89bc-14388a92c001/kube-rbac-proxy/0.log"
Nov 03 08:46:39.205542 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.205526    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4sbrg_60772c10-1acb-489a-89bc-14388a92c001/init-textfile/0.log"
Nov 03 08:46:39.428041 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.428014    5070 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"]
Nov 03 08:46:39.428533 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.428514    5070 cpu_manager.go:414] "containerMap: removing container" podUID="f5bf12cb-42ae-44d1-9bbc-dea3c634690a" containerName="cleanup"
Nov 03 08:46:39.428606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.428534    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5bf12cb-42ae-44d1-9bbc-dea3c634690a" containerName="cleanup"
Nov 03 08:46:39.428606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.428551    5070 cpu_manager.go:414] "containerMap: removing container" podUID="f5bf12cb-42ae-44d1-9bbc-dea3c634690a" containerName="cleanup"
Nov 03 08:46:39.428606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.428556    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5bf12cb-42ae-44d1-9bbc-dea3c634690a" containerName="cleanup"
Nov 03 08:46:39.428606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.428582    5070 cpu_manager.go:414] "containerMap: removing container" podUID="f5bf12cb-42ae-44d1-9bbc-dea3c634690a" containerName="cleanup"
Nov 03 08:46:39.428606 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.428588    5070 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5bf12cb-42ae-44d1-9bbc-dea3c634690a" containerName="cleanup"
Nov 03 08:46:39.428801 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.428668    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5bf12cb-42ae-44d1-9bbc-dea3c634690a" containerName="cleanup"
Nov 03 08:46:39.428801 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.428684    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5bf12cb-42ae-44d1-9bbc-dea3c634690a" containerName="cleanup"
Nov 03 08:46:39.428801 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.428690    5070 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5bf12cb-42ae-44d1-9bbc-dea3c634690a" containerName="cleanup"
Nov 03 08:46:39.431726 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.431711    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"
Nov 03 08:46:39.436876 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.436856    5070 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lsk8v\"/\"default-dockercfg-lfs26\""
Nov 03 08:46:39.440093 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.440070    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lsk8v\"/\"kube-root-ca.crt\""
Nov 03 08:46:39.440209 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.440142    5070 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lsk8v\"/\"openshift-service-ca.crt\""
Nov 03 08:46:39.443431 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.443409    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"]
Nov 03 08:46:39.518702 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.518681    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe7be472-5793-410a-94f5-32c487e21aa7-sys\") pod \"perf-node-gather-daemonset-j87fl\" (UID: \"fe7be472-5793-410a-94f5-32c487e21aa7\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"
Nov 03 08:46:39.518844 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.518729    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe7be472-5793-410a-94f5-32c487e21aa7-lib-modules\") pod \"perf-node-gather-daemonset-j87fl\" (UID: \"fe7be472-5793-410a-94f5-32c487e21aa7\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"
Nov 03 08:46:39.518900 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.518846    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fe7be472-5793-410a-94f5-32c487e21aa7-podres\") pod \"perf-node-gather-daemonset-j87fl\" (UID: \"fe7be472-5793-410a-94f5-32c487e21aa7\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"
Nov 03 08:46:39.518945 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.518905    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fe7be472-5793-410a-94f5-32c487e21aa7-proc\") pod \"perf-node-gather-daemonset-j87fl\" (UID: \"fe7be472-5793-410a-94f5-32c487e21aa7\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"
Nov 03 08:46:39.519003 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.518949    5070 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ntvb\" (UniqueName: \"kubernetes.io/projected/fe7be472-5793-410a-94f5-32c487e21aa7-kube-api-access-6ntvb\") pod \"perf-node-gather-daemonset-j87fl\" (UID: \"fe7be472-5793-410a-94f5-32c487e21aa7\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"
Nov 03 08:46:39.620199 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.620176    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fe7be472-5793-410a-94f5-32c487e21aa7-podres\") pod \"perf-node-gather-daemonset-j87fl\" (UID: \"fe7be472-5793-410a-94f5-32c487e21aa7\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"
Nov 03 08:46:39.620307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.620216    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fe7be472-5793-410a-94f5-32c487e21aa7-proc\") pod \"perf-node-gather-daemonset-j87fl\" (UID: \"fe7be472-5793-410a-94f5-32c487e21aa7\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"
Nov 03 08:46:39.620307 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.620237    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ntvb\" (UniqueName: \"kubernetes.io/projected/fe7be472-5793-410a-94f5-32c487e21aa7-kube-api-access-6ntvb\") pod \"perf-node-gather-daemonset-j87fl\" (UID: \"fe7be472-5793-410a-94f5-32c487e21aa7\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"
Nov 03 08:46:39.620390 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.620315    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fe7be472-5793-410a-94f5-32c487e21aa7-proc\") pod \"perf-node-gather-daemonset-j87fl\" (UID: \"fe7be472-5793-410a-94f5-32c487e21aa7\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"
Nov 03 08:46:39.620390 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.620337    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe7be472-5793-410a-94f5-32c487e21aa7-sys\") pod \"perf-node-gather-daemonset-j87fl\" (UID: \"fe7be472-5793-410a-94f5-32c487e21aa7\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"
Nov 03 08:46:39.620390 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.620364    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fe7be472-5793-410a-94f5-32c487e21aa7-sys\") pod \"perf-node-gather-daemonset-j87fl\" (UID: \"fe7be472-5793-410a-94f5-32c487e21aa7\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"
Nov 03 08:46:39.620390 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.620317    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fe7be472-5793-410a-94f5-32c487e21aa7-podres\") pod \"perf-node-gather-daemonset-j87fl\" (UID: \"fe7be472-5793-410a-94f5-32c487e21aa7\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"
Nov 03 08:46:39.620532 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.620400    5070 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe7be472-5793-410a-94f5-32c487e21aa7-lib-modules\") pod \"perf-node-gather-daemonset-j87fl\" (UID: \"fe7be472-5793-410a-94f5-32c487e21aa7\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"
Nov 03 08:46:39.620532 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.620492    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fe7be472-5793-410a-94f5-32c487e21aa7-lib-modules\") pod \"perf-node-gather-daemonset-j87fl\" (UID: \"fe7be472-5793-410a-94f5-32c487e21aa7\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"
Nov 03 08:46:39.627490 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.627459    5070 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ntvb\" (UniqueName: \"kubernetes.io/projected/fe7be472-5793-410a-94f5-32c487e21aa7-kube-api-access-6ntvb\") pod \"perf-node-gather-daemonset-j87fl\" (UID: \"fe7be472-5793-410a-94f5-32c487e21aa7\") " pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"
Nov 03 08:46:39.722623 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.722563    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-l99g7_bcbcf547-bdcb-46f9-bd2c-9164868a80ec/prometheus-operator/0.log"
Nov 03 08:46:39.739914 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.739891    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-l99g7_bcbcf547-bdcb-46f9-bd2c-9164868a80ec/kube-rbac-proxy/0.log"
Nov 03 08:46:39.745837 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.745819    5070 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"
Nov 03 08:46:39.764930 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.764890    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-j5h2j_22591a72-cf89-4406-8e20-82fd6d0f6e6a/prometheus-operator-admission-webhook/1.log"
Nov 03 08:46:39.765141 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:39.765124    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-j5h2j_22591a72-cf89-4406-8e20-82fd6d0f6e6a/prometheus-operator-admission-webhook/0.log"
Nov 03 08:46:40.022389 ip-10-0-140-123 kubenswrapper[5070]: W1103 08:46:40.022360    5070 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfe7be472_5793_410a_94f5_32c487e21aa7.slice/crio-a9bc67f1cbf4e98bf5dd13757b9abf89835d2d6364c94ff23398ca1536e83e82 WatchSource:0}: Error finding container a9bc67f1cbf4e98bf5dd13757b9abf89835d2d6364c94ff23398ca1536e83e82: Status 404 returned error can't find the container with id a9bc67f1cbf4e98bf5dd13757b9abf89835d2d6364c94ff23398ca1536e83e82
Nov 03 08:46:40.025539 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:40.025517    5070 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"]
Nov 03 08:46:40.051014 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:40.050985    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl" event={"ID":"fe7be472-5793-410a-94f5-32c487e21aa7","Type":"ContainerStarted","Data":"a9bc67f1cbf4e98bf5dd13757b9abf89835d2d6364c94ff23398ca1536e83e82"}
Nov 03 08:46:41.058439 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:41.058397    5070 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl" event={"ID":"fe7be472-5793-410a-94f5-32c487e21aa7","Type":"ContainerStarted","Data":"e7bebb1512a29e08e8050e7541a5fad40f72484e3f8052efc3ebdc4774529a93"}
Nov 03 08:46:41.058837 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:41.058558    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"
Nov 03 08:46:41.072592 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:41.072550    5070 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl" podStartSLOduration=2.072538084 podStartE2EDuration="2.072538084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 08:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 08:46:41.070856627 +0000 UTC m=+3199.185320558" watchObservedRunningTime="2026-11-03 08:46:41.072538084 +0000 UTC m=+3199.187002009"
Nov 03 08:46:41.152805 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:41.152741    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-grgjj_751205b0-8312-4ea6-920c-9517e19566f2/networking-console-plugin/0.log"
Nov 03 08:46:41.731536 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:41.731446    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/4.log"
Nov 03 08:46:41.737101 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:41.737080    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-5qpnb_d83648de-81dd-4160-8fa9-06d36b38d9de/console-operator/3.log"
Nov 03 08:46:42.190981 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:42.190952    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-58b4bd6669-gzcxn_873e56e5-1f64-4eec-a578-bf4ddd072869/console/0.log"
Nov 03 08:46:42.234981 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:42.234957    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-qmpgn_5400462b-2298-41e2-b44b-caa2f3a1e496/download-server/0.log"
Nov 03 08:46:42.770343 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:42.770315    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-zhh6g_3cb721ff-442b-41b7-8287-220dbfbb41da/volume-data-source-validator/0.log"
Nov 03 08:46:43.769317 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:43.769283    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-46l5d_279b3dd8-58be-43a1-9994-59fa434409f5/dns/0.log"
Nov 03 08:46:43.788543 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:43.788517    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-46l5d_279b3dd8-58be-43a1-9994-59fa434409f5/kube-rbac-proxy/0.log"
Nov 03 08:46:43.914708 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:43.914682    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-drdft_f142ef3a-aa21-48eb-a403-8d51d843c72d/dns-node-resolver/0.log"
Nov 03 08:46:44.559255 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:44.559220    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7d9549b467-hqhvz_5211e023-fa82-4010-939e-89afebb9e6b8/registry/0.log"
Nov 03 08:46:44.562088 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:44.562033    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7d9549b467-hqhvz_5211e023-fa82-4010-939e-89afebb9e6b8/registry/1.log"
Nov 03 08:46:44.647081 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:44.647053    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w9c2l_67ff6855-c974-451d-b582-679e7f26e98d/node-ca/0.log"
Nov 03 08:46:46.618416 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:46.618385    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cf769c4c7-l2kwc_0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f/kube-auth-proxy/0.log"
Nov 03 08:46:46.679815 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:46.679790    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-87snj_0c9ad9ff-3d0f-4994-829c-81fe102cc3de/istio-proxy/0.log"
Nov 03 08:46:46.708579 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:46.708537    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-8rcwm_a86a6470-9786-4e2d-8668-62ce0dbbb596/istio-proxy/0.log"
Nov 03 08:46:46.828089 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:46.828068    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-ln9hl_c4c16356-5579-4aac-aec8-c72808dbb1a2/istio-proxy/0.log"
Nov 03 08:46:47.026531 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:47.026455    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79698c9d55-m2lbh_ce890601-127d-4995-9487-635e7aab04f8/router/0.log"
Nov 03 08:46:47.074731 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:47.074707    5070 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lsk8v/perf-node-gather-daemonset-j87fl"
Nov 03 08:46:49.108063 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:49.108023    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6ldkm_78b91dbb-d9e9-4524-8827-d74c8c0625d0/serve-healthcheck-canary/0.log"
Nov 03 08:46:49.727827 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:49.727797    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-crn8p_c620e360-6548-4b0c-ba9a-a9150cf9939c/insights-operator/0.log"
Nov 03 08:46:49.730826 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:49.730792    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-crn8p_c620e360-6548-4b0c-ba9a-a9150cf9939c/insights-operator/1.log"
Nov 03 08:46:49.920077 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:49.920047    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wd497_6731c200-9eda-44f2-aa13-2b4a8fbda85e/kube-rbac-proxy/0.log"
Nov 03 08:46:49.942411 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:49.942385    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wd497_6731c200-9eda-44f2-aa13-2b4a8fbda85e/exporter/0.log"
Nov 03 08:46:49.967474 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:49.967451    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wd497_6731c200-9eda-44f2-aa13-2b4a8fbda85e/extractor/0.log"
Nov 03 08:46:50.011825 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:50.011749    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-kfr2h-k95nf_2c197040-3bfe-416c-be23-5d5b67678173/insights-gathering/0.log"
Nov 03 08:46:50.700554 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:50.700520    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cf769c4c7-l2kwc_0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f/kube-auth-proxy/0.log"
Nov 03 08:46:50.744386 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:50.744335    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-87snj_0c9ad9ff-3d0f-4994-829c-81fe102cc3de/istio-proxy/0.log"
Nov 03 08:46:50.760386 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:50.760361    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-8rcwm_a86a6470-9786-4e2d-8668-62ce0dbbb596/istio-proxy/0.log"
Nov 03 08:46:50.911983 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:50.911953    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-ln9hl_c4c16356-5579-4aac-aec8-c72808dbb1a2/istio-proxy/0.log"
Nov 03 08:46:51.036404 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:51.036331    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79698c9d55-m2lbh_ce890601-127d-4995-9487-635e7aab04f8/router/0.log"
Nov 03 08:46:52.050198 ip-10-0-140-123 kubenswrapper[5070]: E1103 08:46:52.050162    5070 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-wd497" containerName="extractor"
Nov 03 08:46:52.689251 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:52.689217    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6dd9d888fc-qjx65_9ecd3f55-84e0-4321-8d19-900388b6a91f/manager/1.log"
Nov 03 08:46:52.723249 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:52.723201    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6dd9d888fc-qjx65_9ecd3f55-84e0-4321-8d19-900388b6a91f/manager/0.log"
Nov 03 08:46:52.745140 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:52.745120    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6dd9d888fc-qjx65_9ecd3f55-84e0-4321-8d19-900388b6a91f/copy-manifests/0.log"
Nov 03 08:46:55.110835 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:55.110803    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6cf769c4c7-l2kwc_0c9fa25a-f905-4b0e-aa09-24c2a0a1cf8f/kube-auth-proxy/0.log"
Nov 03 08:46:55.155231 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:55.155198    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-87snj_0c9ad9ff-3d0f-4994-829c-81fe102cc3de/istio-proxy/0.log"
Nov 03 08:46:55.175563 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:55.175538    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-8rcwm_a86a6470-9786-4e2d-8668-62ce0dbbb596/istio-proxy/0.log"
Nov 03 08:46:55.271801 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:55.271770    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-ln9hl_c4c16356-5579-4aac-aec8-c72808dbb1a2/istio-proxy/0.log"
Nov 03 08:46:55.407732 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:55.407656    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-79698c9d55-m2lbh_ce890601-127d-4995-9487-635e7aab04f8/router/0.log"
Nov 03 08:46:55.485953 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:55.485927    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-mdwfw_0d26093c-a60a-4cd8-9d43-8ecded62d895/server/1.log"
Nov 03 08:46:55.486104 ip-10-0-140-123 kubenswrapper[5070]: I1103 08:46:55.485996    5070 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-mdwfw_0d26093c-a60a-4cd8-9d43-8ecded62d895/server/0.log"